Skip to content

cache

A three tier cache in front of the mapped instrument tables.

Instrument resolution is moving into the hot paths. The portfolio construction models resolve every holding and position on every refresh, and the execution models resolve an instrument on every order, so a database round trip in that path is the thing to remove. Measured on this machine, a process dictionary answers in 0.18 microseconds, a pipelined Redis read in 3.3, and a Postgres resolution in about a thousand. The read order follows that ordering exactly: this process's own memory, then Redis, then Postgres, with each tier filling in the ones above it on a miss.

Postgres remains the system of record. Nothing here holds anything that cannot be rebuilt from it, so flushing Redis costs one query per instrument actually touched and nothing else, and an unreachable Redis is a miss rather than an error.

Only the current mapping date is cached. A question about a past date goes straight to the existing queries in resolution.py and caches nothing, because a backtest asking about March is not a hot path and caching every historical date would multiply the memory for no benefit.

Five classes divide the work. MappingRedisTier owns the three Redis hashes, both reading and writing every one of them, so the shape written and the shape read cannot drift apart. MappingPostgresTier owns every statement, so the tie-break ordering that the cached answer and the database answer must share is written once. ThreeTierLookup holds the walk from one tier to the next, and its three subclasses supply the reads and writes for the three questions that get asked. MappingCache is what a caller holds, and it owns the engine so that no method needs one passed to it.

Classes:

Name Description
MappingRedisConnection

One lazily opened Redis client for the mapping cache, which reports unavailability rather than raising.
MappingRedisTier

Reads and writes the three Redis hashes that hold one day's instrument mapping.
MappingPostgresTier

Every statement the cache and the warm run against the mapped instrument tables.
ThreeTierLookup

The walk from this process's memory to Redis to Postgres, shared by the three questions the cache answers.
InstrumentIdentityLookup

Instrument id to that instrument's identity.
TokenCandidateLookup

One broker's token to every identity that token carries, for one broker.
OrderHandleLookup

Instrument id to every broker's order handle for that instrument.
MappingCache

A three tier cache in front of the mapped instrument tables, holding its own engine.

Functions:

Name Description
as_text

Decode one value Redis returned, which is bytes unless the client was built to decode responses.

MappingRedisConnection 

MappingRedisConnection(client=None)

One lazily opened Redis client for the mapping cache, which reports unavailability rather than raising.

Redis here sits in front of data Postgres still holds, so an unreachable Redis has to degrade into a cache miss rather than into an error on an order being placed. A caller asks for the client on every use and checks the answer for None, rather than holding it, so a Redis that went away between two look-ups is noticed at the second.

This does not use get_cache from utilities.configurations, and the difference is deliberate rather than an oversight. That client is built for long lived connections: it has no socket_timeout, so a blocking read can sit through an idle period without raising, and it leaves redis-py's default retry in place. Both are right for a connection held silent overnight and wrong here, where a refused port would take about six seconds to be declared refused and the six seconds would land on whichever look-up happened to be first. Disabling the retry and setting both timeouts explicitly reaches the same conclusion in under a millisecond. The address still comes from redis_configuration, so there remains one source of truth for where Redis is; only how this client waits on it differs.

Attributes:

Name Type Description
RETRY_SECONDS float

How long a failed connection is remembered before another attempt.
CONNECT_TIMEOUT_SECONDS float

How long one connection attempt may take.
SOCKET_TIMEOUT_SECONDS float

How long one command may take once connected.
failed_connections int

Attempts that did not succeed, counted so a Redis nobody can reach is visible rather than silent.

Build the connection, optionally around a client the caller already has.

Parameters:

Name Type Description Default
client Redis | None

An already connected client, which is how a test or a caller with its own pool supplies one. None opens a client on first use.

 None

Returns:

Type Description
None

This method returns nothing.

Example programs

examples/stock_brokers/instruments/mapping/utilities/cache/MappingRedisConnection/example_1_wrapping_a_client_you_already_have.py
"""Hands the mapping cache a Redis client the caller already holds, and drops it after a failure.

A `MappingRedisConnection` normally opens its own Redis client the first time `client()` is asked for one. A caller that already has a client, such as a test or a process with its own connection pool, can pass it in instead, and `client()` then returns that same object every time without opening anything.

This program passes a small stand-in client, which answers `ping` and nothing else, so no Redis server is needed. It then calls `forget()`, which is what the cache does after a Redis command fails. Notice that the next `client()` call answers None rather than trying again straight away: the connection waits `RETRY_SECONDS` before its next attempt, so a Redis that has gone away costs one failure a minute rather than one on every look-up.

Run it from the project root:

    python examples/stock_brokers/instruments/mapping/utilities/cache/MappingRedisConnection/example_1_wrapping_a_client_you_already_have.py
"""

from stock_brokers.instruments.mapping.utilities.cache import (
    MappingRedisConnection,
)


class StandInClient:
    """A stand-in for a connected Redis client that only answers `ping`.

    Attributes:
        name (str): A label printed in place of the client's address.
    """

    def __init__(self, name):
        """Builds the stand-in with a label.

        Args:
            name (str): A label printed in place of the client's address.

        Returns:
            None: This method returns nothing.
        """
        self.name = name

    def ping(self):
        """Answers a ping as a connected Redis would.

        Returns:
            bool: Always True.
        """
        return True


class WrappingAClientExample:
    """Wraps a stand-in client in a mapping connection, then forgets it.

    Attributes:
        stand_in_client (StandInClient): The client the caller already holds.
        connection (MappingRedisConnection): The connection being shown.
    """

    def __init__(self):
        """Builds the connection around the stand-in client.

        Returns:
            None: This method returns nothing.
        """
        self.stand_in_client = StandInClient('the caller pool client')
        self.connection = MappingRedisConnection(self.stand_in_client)

    def run(self):
        """Asks for the client twice, forgets it, and asks again.

        Returns:
            None: This method returns nothing.
        """
        first = self.connection.client()
        second = self.connection.client()
        print(f'First client: {first.name}')
        print(f'Same object both times: {first is second}')
        print(f'Answers ping: {first.ping()}')
        print(f'Failed connections so far: {self.connection.failed_connections}')
        self.connection.forget()
        print('After forget(), a command failed and the client was dropped.')
        print(f'Client straight after forget(): {self.connection.client()}')
        print(f'Seconds before the next attempt: {MappingRedisConnection.RETRY_SECONDS}')
        print(f'Failed connections so far: {self.connection.failed_connections}')


if __name__ == '__main__':
    WrappingAClientExample().run()

It prints:

First client: the caller pool client
Same object both times: True
Answers ping: True
Failed connections so far: 0
After forget(), a command failed and the client was dropped.
Client straight after forget(): None
Seconds before the next attempt: 60.0
Failed connections so far: 0
examples/stock_brokers/instruments/mapping/utilities/cache/MappingRedisConnection/example_2_unreachable_redis_is_a_miss.py
"""Shows that a Redis nobody can reach gives back None quickly instead of raising.

When a `MappingRedisConnection` is built without a client, `client()` opens one from `redis_configuration` and pings it. The client has redis-py's retry switched off and short timeouts, so a refused port is noticed in well under a second, the failure is counted in `failed_connections`, and `client()` answers None. Every caller of the mapping cache checks for None and treats it as a cache miss, so an order still resolves through Postgres.

This program points `redis_configuration` at port 9 on this machine, where nothing listens, so it never touches a real Redis wherever it is run. Notice that the second call within the retry window answers None without a second attempt, because the failure is remembered for `RETRY_SECONDS`, and that the three timeout settings are the ones the class declares.

Run it from the project root:

    python examples/stock_brokers/instruments/mapping/utilities/cache/MappingRedisConnection/example_2_unreachable_redis_is_a_miss.py
"""

from stock_brokers.instruments.mapping.utilities.cache import (
    MappingRedisConnection,
)
from utilities.configurations import redis_configuration


class UnreachableRedisExample:
    """Asks for a client from a Redis address where nothing listens.

    Attributes:
        connection (MappingRedisConnection): The connection being shown.
    """

    def __init__(self):
        """Points the Redis configuration at a closed port and builds the connection.

        Returns:
            None: This method returns nothing.
        """
        redis_configuration['host'] = '127.0.0.1'
        redis_configuration['port'] = 9
        self.connection = MappingRedisConnection()

    def run(self):
        """Asks for the client twice and prints what came back and what was counted.

        Returns:
            None: This method returns nothing.
        """
        print(f'Connect timeout: {MappingRedisConnection.CONNECT_TIMEOUT_SECONDS} seconds')
        print(f'Command timeout: {MappingRedisConnection.SOCKET_TIMEOUT_SECONDS} seconds')
        print(f'Retry window: {MappingRedisConnection.RETRY_SECONDS} seconds')
        first = self.connection.client()
        print(f'First attempt: {first}')
        print(f'Failed connections: {self.connection.failed_connections}')
        second = self.connection.client()
        print(f'Second attempt inside the retry window: {second}')
        print(f'Failed connections: {self.connection.failed_connections}')
        self.connection.forget()
        print(f'After forget(): {self.connection.client()}')
        print(f'Failed connections: {self.connection.failed_connections}')


if __name__ == '__main__':
    UnreachableRedisExample().run()

It prints:

Connect timeout: 0.5 seconds
Command timeout: 1.0 seconds
Retry window: 60.0 seconds
First attempt: None
Failed connections: 1
Second attempt inside the retry window: None
Failed connections: 1
After forget(): None
Failed connections: 1

Methods:

Name Description
client

The connected Redis client, or None when Redis cannot be reached right now.
forget

Drop the client after a failure, so the next look-up reconnects rather than reusing a broken socket.
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
def __init__(self, client=None):
    """
    Build the connection, optionally around a client the caller already has.

    Args:
        client (redis.Redis | None): An already connected client, which is how a test or a
            caller with its own pool supplies one. None opens a client on first use.

    Returns:
        None: This method returns nothing.
    """
    self.failed_connections = 0
    self._client = client
    self._unavailable_until = 0.0

client 

client()

The connected Redis client, or None when Redis cannot be reached right now.

Returns:

Type Description
Redis | None

A connected client, or None when Redis could not be reached and the retry window has not yet passed.

Example usage

From example_1_wrapping_a_client_you_already_have.py
first = self.connection.client()
From example_1_wrapping_a_client_you_already_have.py
second = self.connection.client()
From example_1_wrapping_a_client_you_already_have.py
print(f'Client straight after forget(): {self.connection.client()}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
def client(self):
    """
    The connected Redis client, or None when Redis cannot be reached right now.

    Returns:
        redis.Redis | None: A connected client, or None when Redis could not be reached and the
            retry window has not yet passed.
    """
    if self._client is not None:
        return self._client
    if time.monotonic() < self._unavailable_until:
        return None
    try:
        client = redis.Redis(
            host=redis_configuration["host"],
            port=redis_configuration["port"],
            db=redis_configuration["db"],
            username=redis_configuration["username"],
            password=redis_configuration["password"],
            socket_connect_timeout=self.CONNECT_TIMEOUT_SECONDS,
            socket_timeout=self.SOCKET_TIMEOUT_SECONDS,
            retry=Retry(NoBackoff(), 0),
        )
        client.ping()
    except redis.RedisError:
        self.failed_connections += 1
        self._unavailable_until = time.monotonic() + self.RETRY_SECONDS
        return None
    self._client = client
    return self._client

forget 

forget()

Drop the client after a failure, so the next look-up reconnects rather than reusing a broken socket.

Returns:

Type Description
None

This method returns nothing.

Example usage

From example_1_wrapping_a_client_you_already_have.py
self.connection.forget()
From example_2_unreachable_redis_is_a_miss.py
self.connection.forget()
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
134
135
136
137
138
139
140
141
142
def forget(self):
    """
    Drop the client after a failure, so the next look-up reconnects rather than reusing a broken socket.

    Returns:
        None: This method returns nothing.
    """
    self._client = None
    self._unavailable_until = time.monotonic() + self.RETRY_SECONDS

MappingRedisTier 

MappingRedisTier(connection=None)

Reads and writes the three Redis hashes that hold one day's instrument mapping.

Every read and every write of these hashes goes through this class, including the daily warm's. That is the whole point of it. If the reader and the warm each encoded the instrument-keyed entry, a four field order handle, on their own, a change made to one and not the other would leave Redis holding a shape that decodes successfully into something the caller cannot use. With one class owning both directions there is nowhere for the two shapes to disagree.

Attributes:

Name Type Description
KEY_PREFIX str

The prefix every key in this cache shares, tables.MAPPING_CACHE_PREFIX: unified:catalogue:.
connection MappingRedisConnection

The connection the hashes are read and written through.

Build the tier around a Redis connection.

Parameters:

Name Type Description Default
connection MappingRedisConnection | None

The connection to use. None builds one.

 None

Returns:

Type Description
None

This function returns nothing.

Example programs

examples/stock_brokers/instruments/mapping/utilities/cache/MappingRedisTier/example_1_keys_and_encodings.py
"""Prints every Redis key the mapping cache uses for one day, and how each value is encoded and decoded.

A `MappingRedisTier` is the only code that reads or writes the mapping cache's Redis keys, so the names and value layouts it builds are the whole contract between the daily warm and the readers. This program asks it for every key of one mapping date and then round-trips one identity, one set of order handles, one set of additional attributes, one contract size decision, one pair of seen dates and one catalogue member through the encoders and decoders.

None of these methods talk to Redis, so the tier is built around a connection whose client is a stand-in that is never used. Notice three things in the output. The strike in a catalogue member is zero-padded so that sorting the text sorts the strikes numerically. A catalogue prefix stops at the first field that is not given, so a name alone matches every contract under it. And a value held in an older, narrower shape decodes to None, which the cache treats as a miss rather than as a wrong answer. `seconds_until_midnight()` depends on the clock, so the program prints only that it is at least sixty.

Run it from the project root:

    python examples/stock_brokers/instruments/mapping/utilities/cache/MappingRedisTier/example_1_keys_and_encodings.py
"""

import datetime
import decimal

from stock_brokers.instruments.mapping.utilities.cache import (
    MappingRedisConnection,
    MappingRedisTier,
)


class UnusedClient:
    """A stand-in Redis client that none of the methods shown here ever calls."""


class KeysAndEncodingsExample:
    """Prints the tier's keys for one date and round-trips each kind of stored value.

    Attributes:
        tier (MappingRedisTier): The tier being shown.
        mapping_date (datetime.date): The mapping date the keys are built for.
        option_identity (dict): The identity of one NIFTY option.
        equity_identity (dict): The identity of one NSE equity.
    """

    def __init__(self):
        """Builds the tier and the two identities.

        Returns:
            None: This method returns nothing.
        """
        self.tier = MappingRedisTier(MappingRedisConnection(UnusedClient()))
        self.mapping_date = datetime.date(2026, 9, 30)
        self.option_identity = {
            'instrument_id': '5f0c7a8e-3b1d-4c62-9e4a-2d8b7f1c0a31',
            'exchange': 'nse',
            'segment': 'nse_equity_index_options',
            'shape': 'option',
            'symbol': None,
            'underlying_symbol': 'NIFTY',
            'expiry_date': datetime.date(2026, 10, 6),
            'strike_price': decimal.Decimal('25000'),
            'option_type': 'CE',
            'mapping_date': self.mapping_date,
        }
        self.equity_identity = {
            'instrument_id': '0a6e2d41-9c7f-4b58-8d13-6f2e4a9b7c05',
            'exchange': 'nse',
            'segment': 'nse_equities',
            'shape': 'security',
            'symbol': 'INFY',
            'underlying_symbol': None,
            'expiry_date': None,
            'strike_price': None,
            'option_type': None,
            'mapping_date': self.mapping_date,
        }

    def print_keys(self):
        """Prints every key the tier builds for the mapping date.

        Returns:
            None: This method returns nothing.
        """
        print(f'Key prefix: {self.tier.KEY_PREFIX}')
        print(f'current_date_key: {self.tier.current_date_key()}')
        print(f'warm_identifier_key: {self.tier.warm_identifier_key()}')
        print(f'identity_key: {self.tier.identity_key(self.mapping_date)}')
        print(f'tokens_key: {self.tier.tokens_key(self.mapping_date, "zerodha")}')
        print(f'order_handles_key: {self.tier.order_handles_key(self.mapping_date)}')
        print(f'contract_sizes_key: {self.tier.contract_sizes_key(self.mapping_date)}')
        print(f'underlyings_key: {self.tier.underlyings_key(self.mapping_date)}')
        print(f'additional_attributes_key: {self.tier.additional_attributes_key(self.mapping_date)}')
        print(f'segments_key: {self.tier.segments_key(self.mapping_date)}')
        print(f'catalogue_key: {self.tier.catalogue_key(self.mapping_date, "nse_equities")}')
        print(f'names_key: {self.tier.names_key(self.mapping_date, "nse_equity_index_options")}')
        print(f'seen_key: {self.tier.seen_key(self.mapping_date)}')

    def print_catalogue_encoding(self):
        """Prints the catalogue names, members and prefixes of the two identities.

        Returns:
            None: This method returns nothing.
        """
        print(f'Catalogue name of the equity: {self.tier.catalogue_name(self.equity_identity)}')
        print(f'Catalogue name of the option: {self.tier.catalogue_name(self.option_identity)}')
        equity_member = self.tier.encode_catalogue_member(self.equity_identity)
        option_member = self.tier.encode_catalogue_member(self.option_identity)
        print(f'Equity member: {equity_member}')
        print(f'Option member: {option_member}')
        print(f'Instrument id read back from the option member: {self.tier.decode_catalogue_member(option_member)}')
        print(f'Prefix for a name alone: {self.tier.catalogue_prefix("NIFTY")}')
        expiry_prefix = self.tier.catalogue_prefix('NIFTY', datetime.date(2026, 10, 6))
        print(f'Prefix for a name and an expiry: {expiry_prefix}')
        whole_prefix = self.tier.catalogue_prefix(
            'NIFTY',
            datetime.date(2026, 10, 6),
            decimal.Decimal('25000'),
            'CE',
        )
        print(f'Prefix for a whole option: {whole_prefix}')
        print(f'The option member starts with it: {option_member.startswith(whole_prefix)}')

    def print_value_encodings(self):
        """Round-trips an identity, order handles, attributes, a contract size and seen dates.

        Returns:
            None: This method returns nothing.
        """
        encoded_identity = self.tier.encode_identity(self.option_identity)
        print(f'Encoded identity: {encoded_identity}')
        decoded_identity = self.tier.decode_identity(encoded_identity)
        print(f'Decoded expiry and strike: {decoded_identity["expiry_date"]!r} {decoded_identity["strike_price"]!r}')
        print(f'Decoding text that is not JSON: {self.tier.decode_identity("not json")}')
        handles = {
            'zerodha': {
                'broker_token': '12053250',
                'order_symbol': 'NIFTY26O0625000CE',
                'lot_size': '75',
                'tick_size': '0.05',
            },
        }
        encoded_handles = self.tier.encode_order_handles(handles)
        print(f'Encoded order handles: {encoded_handles}')
        print(f'Decoded order handles: {self.tier.decode_order_handles(encoded_handles)}')
        older_shape = '{"zerodha": "12053250"}'
        print(f'Decoding a bare token in the older shape: {self.tier.decode_order_handles(older_shape)}')
        attributes = {
            'dhan': {
                'freeze_quantity': '1800',
                'isin': None,
            },
        }
        encoded_attributes = self.tier.encode_additional_attributes(attributes)
        print(f'Encoded additional attributes: {encoded_attributes}')
        print(f'Decoded additional attributes: {self.tier.decode_additional_attributes(encoded_attributes)}')
        print(f'Decoding a list: {self.tier.decode_additional_attributes("[1, 2]")}')
        size = self.tier.encode_contract_size(decimal.Decimal('1000.000'), 'confirmed', True)
        print(f'Encoded contract size: {size}')
        no_source = self.tier.encode_contract_size(None, 'no_source', False)
        print(f'Encoded contract size with no source: {no_source}')
        seen = self.tier.encode_seen(datetime.date(2024, 1, 2), self.mapping_date)
        print(f'Encoded seen dates: {seen}')
        print(f'Keys live at least a minute: {self.tier.seconds_until_midnight() >= 60}')

    def run(self):
        """Prints the keys, then the catalogue encoding, then the value encodings.

        Returns:
            None: This method returns nothing.
        """
        self.print_keys()
        self.print_catalogue_encoding()
        self.print_value_encodings()


if __name__ == '__main__':
    KeysAndEncodingsExample().run()

It prints:

Key prefix: unified:catalogue:
current_date_key: unified:catalogue:current_date
warm_identifier_key: unified:catalogue:warm_identifier
identity_key: unified:catalogue:2026-09-30:identity
tokens_key: unified:catalogue:2026-09-30:tokens:zerodha
order_handles_key: unified:catalogue:2026-09-30:order_handles
contract_sizes_key: unified:catalogue:2026-09-30:contract_sizes
underlyings_key: unified:catalogue:2026-09-30:underlyings
additional_attributes_key: unified:catalogue:2026-09-30:additional_attributes
segments_key: unified:catalogue:2026-09-30:segments
catalogue_key: unified:catalogue:2026-09-30:catalogue:nse_equities
names_key: unified:catalogue:2026-09-30:names:nse_equity_index_options
seen_key: unified:catalogue:2026-09-30:seen
Catalogue name of the equity: INFY
Catalogue name of the option: NIFTY
Equity member: INFY||||0a6e2d41-9c7f-4b58-8d13-6f2e4a9b7c05
Option member: NIFTY|2026-10-06|00000025000.0000|CE|5f0c7a8e-3b1d-4c62-9e4a-2d8b7f1c0a31
Instrument id read back from the option member: 5f0c7a8e-3b1d-4c62-9e4a-2d8b7f1c0a31
Prefix for a name alone: NIFTY|
Prefix for a name and an expiry: NIFTY|2026-10-06|
Prefix for a whole option: NIFTY|2026-10-06|00000025000.0000|CE|
The option member starts with it: True
Encoded identity: {"instrument_id": "5f0c7a8e-3b1d-4c62-9e4a-2d8b7f1c0a31", "exchange": "nse", "segment": "nse_equity_index_options", "shape": "option", "symbol": null, "underlying_symbol": "NIFTY", "expiry_date": "2026-10-06", "strike_price": "25000", "option_type": "CE", "mapping_date": "2026-09-30"}
Decoded expiry and strike: datetime.date(2026, 10, 6) Decimal('25000')
Decoding text that is not JSON: None
Encoded order handles: {"zerodha": {"broker_token": "12053250", "order_symbol": "NIFTY26O0625000CE", "lot_size": "75", "tick_size": "0.05"}}
Decoded order handles: {'zerodha': {'broker_token': '12053250', 'order_symbol': 'NIFTY26O0625000CE', 'lot_size': '75', 'tick_size': '0.05'}}
Decoding a bare token in the older shape: None
Encoded additional attributes: {"dhan": {"freeze_quantity": "1800", "isin": null}}
Decoded additional attributes: {'dhan': {'freeze_quantity': '1800', 'isin': None}}
Decoding a list: None
Encoded contract size: {"units_per_lot": "1000", "status": "confirmed", "tradeable": true}
Encoded contract size with no source: {"units_per_lot": null, "status": "no_source", "tradeable": false}
Encoded seen dates: 2024-01-02|2026-09-30
Keys live at least a minute: True
examples/stock_brokers/instruments/mapping/utilities/cache/MappingRedisTier/example_2_writing_and_reading_one_day.py
"""Writes one day's mapping into the Redis hashes and reads every part of it back.

This is what the daily warm and the readers do between them. The program publishes the mapping date, writes two identities, one Zerodha token that carries one of them, both instruments' order handles, and the seen dates, underlyings, additional attributes and segment counts that the warm streams in with `write_fields`. It then reads each hash back through the tier's own readers, which is the point of the class: the shape written and the shape read come from the same code.

A small in-memory stand-in replaces the Redis client. It keeps strings and hashes in dictionaries, records each key's expiry, and runs a pipeline's commands when the pipeline is executed, which is all the tier uses. Notice that `read_token_identifiers` gives back instrument ids rather than identities, that an instrument with no resolved underlying is simply absent, and that `has_additional_attributes` is what tells "this instrument has no attributes" apart from "the hash was never warmed". The warm identifier is a new random value on every run, so the program prints only that one was written.

Run it from the project root:

    python examples/stock_brokers/instruments/mapping/utilities/cache/MappingRedisTier/example_2_writing_and_reading_one_day.py
"""

import datetime
import decimal

from stock_brokers.instruments.mapping.utilities.cache import (
    MappingRedisConnection,
    MappingRedisTier,
)


class StandInPipeline:
    """A stand-in for a Redis pipeline that queues commands and runs them on execute.

    Attributes:
        redis_client (StandInRedis): The stand-in client the commands run against.
        queued (list): The queued commands, as (method, arguments) pairs.
    """

    def __init__(self, redis_client):
        """Builds an empty pipeline.

        Args:
            redis_client (StandInRedis): The stand-in client the commands run against.

        Returns:
            None: This method returns nothing.
        """
        self.redis_client = redis_client
        self.queued = []

    def set(self, key, value):
        """Queues a string write.

        Args:
            key (str): The key to write.
            value (str): The value to store.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.set, (key, value)))

    def hset(self, key, mapping):
        """Queues a hash write.

        Args:
            key (str): The hash to write into.
            mapping (dict): The fields to write.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.hset, (key, mapping)))

    def hmget(self, key, fields):
        """Queues a read of several hash fields.

        Args:
            key (str): The hash to read.
            fields (list): The fields to read.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.hmget, (key, fields)))

    def expire(self, key, seconds):
        """Queues an expiry.

        Args:
            key (str): The key to expire.
            seconds (int): How long the key may live.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.expire, (key, seconds)))

    def execute(self):
        """Runs every queued command in order.

        Returns:
            list: Each command's reply, in the order queued.
        """
        replies = []
        for method, arguments in self.queued:
            replies.append(method(*arguments))
        self.queued = []
        return replies


class StandInRedis:
    """A stand-in for a Redis client that keeps strings and hashes in dictionaries.

    Attributes:
        values (dict): Each key's value: a string, or a dict for a hash.
        expiries (dict): Each key's expiry in seconds, as last set.
    """

    def __init__(self):
        """Builds an empty stand-in.

        Returns:
            None: This method returns nothing.
        """
        self.values = {}
        self.expiries = {}

    def pipeline(self, transaction=True):
        """Starts a pipeline.

        Args:
            transaction (bool): Ignored; the stand-in always runs commands in order.

        Returns:
            StandInPipeline: A new pipeline.
        """
        return StandInPipeline(self)

    def get(self, key):
        """Reads a string.

        Args:
            key (str): The key to read.

        Returns:
            str | None: The value, or None when the key is absent.
        """
        return self.values.get(key)

    def set(self, key, value):
        """Writes a string.

        Args:
            key (str): The key to write.
            value (str): The value to store.

        Returns:
            bool: Always True.
        """
        self.values[key] = value
        return True

    def exists(self, key):
        """Counts whether a key exists.

        Args:
            key (str): The key to test.

        Returns:
            int: 1 when the key exists, otherwise 0.
        """
        if key in self.values:
            return 1
        return 0

    def expire(self, key, seconds):
        """Records a key's expiry.

        Args:
            key (str): The key to expire.
            seconds (int): How long the key may live.

        Returns:
            bool: Always True.
        """
        self.expiries[key] = seconds
        return True

    def hset(self, key, mapping):
        """Writes fields into a hash.

        Args:
            key (str): The hash to write into.
            mapping (dict): The fields to write.

        Returns:
            int: The number of fields written.
        """
        stored = self.values.setdefault(key, {})
        stored.update(mapping)
        return len(mapping)

    def hmget(self, key, fields):
        """Reads several fields of a hash.

        Args:
            key (str): The hash to read.
            fields (list): The fields to read.

        Returns:
            list: Each field's value, or None where the field is absent.
        """
        stored = self.values.get(key, {})
        replies = []
        for field in fields:
            replies.append(stored.get(field))
        return replies

    def hgetall(self, key):
        """Reads a whole hash.

        Args:
            key (str): The hash to read.

        Returns:
            dict: The hash's fields, empty when the key is absent.
        """
        return dict(self.values.get(key, {}))


class WritingAndReadingOneDayExample:
    """Writes one mapping date's hashes through the tier and reads them back.

    Attributes:
        redis_client (StandInRedis): The stand-in Redis the tier writes into.
        tier (MappingRedisTier): The tier being shown.
        mapping_date (datetime.date): The mapping date written.
        future_identity (dict): The identity of one NIFTY future.
        index_identity (dict): The identity of the NIFTY 50 index.
    """

    FUTURE_ID = '7c1e9b20-4d3a-4f87-a6b5-0e2d9c8f1a44'
    INDEX_ID = '2b9d4e6f-8a1c-4e35-b7d2-5f0a3c6e9b18'

    def __init__(self):
        """Builds the tier around the stand-in and the two identities.

        Returns:
            None: This method returns nothing.
        """
        self.redis_client = StandInRedis()
        self.tier = MappingRedisTier(MappingRedisConnection(self.redis_client))
        self.mapping_date = datetime.date(2026, 9, 30)
        self.future_identity = {
            'instrument_id': self.FUTURE_ID,
            'exchange': 'nse',
            'segment': 'nse_equity_index_futures',
            'shape': 'future',
            'symbol': None,
            'underlying_symbol': 'NIFTY',
            'expiry_date': datetime.date(2026, 10, 27),
            'strike_price': None,
            'option_type': None,
            'mapping_date': self.mapping_date,
        }
        self.index_identity = {
            'instrument_id': self.INDEX_ID,
            'exchange': 'nse',
            'segment': 'nse_equity_indices',
            'shape': 'security',
            'symbol': 'NIFTY 50',
            'underlying_symbol': None,
            'expiry_date': None,
            'strike_price': None,
            'option_type': None,
            'mapping_date': self.mapping_date,
        }

    def write_day(self):
        """Writes every hash for the mapping date, publishing the date last as the warm does.

        Returns:
            None: This method returns nothing.
        """
        identities = {
            self.FUTURE_ID: self.future_identity,
            self.INDEX_ID: self.index_identity,
        }
        print(f'write_identities: {self.tier.write_identities(self.mapping_date, identities)}')
        candidates_by_token = {
            '13238018': [
                self.future_identity,
            ],
        }
        print(f'write_token_candidates: {self.tier.write_token_candidates(self.mapping_date, "zerodha", candidates_by_token)}')
        handles_by_instrument = {
            self.FUTURE_ID: {
                'zerodha': {
                    'broker_token': '13238018',
                    'order_symbol': 'NIFTY26OCTFUT',
                    'lot_size': '75',
                    'tick_size': '0.1',
                },
                'dhan': {
                    'broker_token': '52168',
                    'order_symbol': None,
                    'lot_size': '75',
                    'tick_size': '0.1',
                },
            },
        }
        print(f'write_order_handles: {self.tier.write_order_handles(self.mapping_date, handles_by_instrument)}')
        expiry_seconds = 3600
        seen_fields = {
            self.FUTURE_ID: self.tier.encode_seen(datetime.date(2026, 7, 1), self.mapping_date),
            self.INDEX_ID: self.tier.encode_seen(datetime.date(2024, 1, 2), self.mapping_date),
        }
        underlying_fields = {
            self.FUTURE_ID: self.INDEX_ID,
        }
        attribute_fields = {
            self.FUTURE_ID: self.tier.encode_additional_attributes({
                'dhan': {
                    'freeze_quantity': '1800',
                },
            }),
        }
        contract_size_fields = {
            self.FUTURE_ID: self.tier.encode_contract_size(decimal.Decimal('75'), 'confirmed', True),
        }
        segment_fields = {
            'nse_equity_index_futures': '1',
            'nse_equity_indices': '1',
        }
        written = 0
        written += self.tier.write_fields(self.tier.seen_key(self.mapping_date), seen_fields, expiry_seconds)
        written += self.tier.write_fields(self.tier.underlyings_key(self.mapping_date), underlying_fields, expiry_seconds)
        written += self.tier.write_fields(self.tier.additional_attributes_key(self.mapping_date), attribute_fields, expiry_seconds)
        written += self.tier.write_fields(self.tier.contract_sizes_key(self.mapping_date), contract_size_fields, expiry_seconds)
        written += self.tier.write_fields(self.tier.segments_key(self.mapping_date), segment_fields, expiry_seconds)
        print(f'write_fields wrote {written} fields in all')
        print(f'write_current_date: {self.tier.write_current_date(self.mapping_date)}')
        print(f'A warm identifier was written: {self.tier.warm_identifier_key() in self.redis_client.values}')

    def read_day(self):
        """Reads every hash back through the tier's readers and prints what came back.

        Returns:
            None: This method returns nothing.
        """
        both = [
            self.FUTURE_ID,
            self.INDEX_ID,
        ]
        print(f'read_current_date: {self.tier.read_current_date()!r}')
        print(f'read_segment_counts: {self.tier.read_segment_counts(self.mapping_date)}')
        identities = self.tier.read_identities(self.mapping_date, both)
        for instrument_identifier in both:
            identity = identities[instrument_identifier]
            print(f'read_identities: {identity["segment"]} {identity["symbol"] or identity["underlying_symbol"]} expiring {identity["expiry_date"]}')
        tokens = [
            '13238018',
            '99999999',
        ]
        print(f'read_token_identifiers: {self.tier.read_token_identifiers(self.mapping_date, "zerodha", tokens)}')
        handles = self.tier.read_order_handles(self.mapping_date, both)
        print(f'read_order_handles for the future: {handles[self.FUTURE_ID]}')
        print(f'Instruments with order handles: {len(handles)}')
        print(f'read_seen: {self.tier.read_seen(self.mapping_date, both)}')
        print(f'read_underlyings: {self.tier.read_underlyings(self.mapping_date, both)}')
        seen, underlyings = self.tier.read_seen_and_underlyings(self.mapping_date, both)
        print(f'read_seen_and_underlyings: {len(seen)} seen, {len(underlyings)} underlying')
        print(f'read_additional_attributes: {self.tier.read_additional_attributes(self.mapping_date, both)}')
        print(f'has_additional_attributes today: {self.tier.has_additional_attributes(self.mapping_date)}')
        yesterday = self.mapping_date - datetime.timedelta(days=1)
        print(f'has_additional_attributes yesterday: {self.tier.has_additional_attributes(yesterday)}')
        print(f'read_segment_counts yesterday: {self.tier.read_segment_counts(yesterday)}')
        print(f'Expiry set on the seen hash: {self.redis_client.expiries[self.tier.seen_key(self.mapping_date)]} seconds')

    def run(self):
        """Writes the day, then reads it back.

        Returns:
            None: This method returns nothing.
        """
        self.write_day()
        self.read_day()


if __name__ == '__main__':
    WritingAndReadingOneDayExample().run()

It prints:

write_identities: True
write_token_candidates: True
write_order_handles: True
write_fields wrote 7 fields in all
write_current_date: True
A warm identifier was written: True
read_current_date: datetime.date(2026, 9, 30)
read_segment_counts: {'nse_equity_index_futures': 1, 'nse_equity_indices': 1}
read_identities: nse_equity_index_futures NIFTY expiring 2026-10-27
read_identities: nse_equity_indices NIFTY 50 expiring None
read_token_identifiers: {'13238018': ['7c1e9b20-4d3a-4f87-a6b5-0e2d9c8f1a44']}
read_order_handles for the future: {'zerodha': {'broker_token': '13238018', 'order_symbol': 'NIFTY26OCTFUT', 'lot_size': '75', 'tick_size': '0.1'}, 'dhan': {'broker_token': '52168', 'order_symbol': None, 'lot_size': '75', 'tick_size': '0.1'}}
Instruments with order handles: 1
read_seen: {'7c1e9b20-4d3a-4f87-a6b5-0e2d9c8f1a44': (datetime.date(2026, 7, 1), datetime.date(2026, 9, 30)), '2b9d4e6f-8a1c-4e35-b7d2-5f0a3c6e9b18': (datetime.date(2024, 1, 2), datetime.date(2026, 9, 30))}
read_underlyings: {'7c1e9b20-4d3a-4f87-a6b5-0e2d9c8f1a44': '2b9d4e6f-8a1c-4e35-b7d2-5f0a3c6e9b18'}
read_seen_and_underlyings: 2 seen, 1 underlying
read_additional_attributes: {'7c1e9b20-4d3a-4f87-a6b5-0e2d9c8f1a44': {'dhan': {'freeze_quantity': '1800'}}}
has_additional_attributes today: True
has_additional_attributes yesterday: False
read_segment_counts yesterday: None
Expiry set on the seen hash: 3600 seconds
examples/stock_brokers/instruments/mapping/utilities/cache/MappingRedisTier/example_3_searching_the_catalogue.py
"""Builds a small option catalogue in Redis and searches it by name, expiry, strike and option type.

The catalogue is one sorted set per segment in which every member has the score zero, so Redis keeps the members in plain text order. Each member is laid out as name, expiry, strike, option type and instrument id, so text order is also the order a person would want to browse contracts in, and every contract under one name, or one expiry, sits in a single range that starts with a known prefix. A second sorted set per segment lists only the distinct names, which is what a search box scans.

This program writes six NIFTY and BANKNIFTY options and their two names with `write_members`, then reads them back four ways: a slice by position, every contract under a prefix, the first contract under each of several prefixes in one round trip, and the list of names. Finally it leaves a stale key from the day before in Redis and calls `clear_other_dates`, which deletes every dated key except the kept date's and the two undated keys.

A small in-memory stand-in replaces the Redis client. It keeps each sorted set as a Python set and sorts it on every read, which gives the same order as Redis for members made of plain text. Notice that the 900 strike sorts before the 1000 strike, which is only true because the tier zero-pads strikes, and that a prefix nothing starts with reads as None in the batched form.

Run it from the project root:

    python examples/stock_brokers/instruments/mapping/utilities/cache/MappingRedisTier/example_3_searching_the_catalogue.py
"""

import datetime
import decimal

from stock_brokers.instruments.mapping.utilities.cache import (
    MappingRedisConnection,
    MappingRedisTier,
)


class StandInPipeline:
    """A stand-in for a Redis pipeline that queues commands and runs them on execute.

    Attributes:
        redis_client (StandInRedis): The stand-in client the commands run against.
        queued (list): The queued commands, as (method, arguments) pairs.
    """

    def __init__(self, redis_client):
        """Builds an empty pipeline.

        Args:
            redis_client (StandInRedis): The stand-in client the commands run against.

        Returns:
            None: This method returns nothing.
        """
        self.redis_client = redis_client
        self.queued = []

    def zadd(self, key, mapping):
        """Queues adding members to a sorted set.

        Args:
            key (str): The sorted set.
            mapping (dict): Each member to its score.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.zadd, (key, mapping)))

    def expire(self, key, seconds):
        """Queues an expiry.

        Args:
            key (str): The key to expire.
            seconds (int): How long the key may live.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.expire, (key, seconds)))

    def zrangebylex(self, key, minimum, maximum, start=None, num=None):
        """Queues a read of a sorted set's members between two text bounds.

        Args:
            key (str): The sorted set.
            minimum (bytes): The lower bound, starting with "[" for inclusive.
            maximum (bytes): The upper bound, starting with "(" for exclusive.
            start (int | None): How many matching members to skip.
            num (int | None): The most members to return.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.zrangebylex, (key, minimum, maximum, start, num)))

    def execute(self):
        """Runs every queued command in order.

        Returns:
            list: Each command's reply, in the order queued.
        """
        replies = []
        for method, arguments in self.queued:
            replies.append(method(*arguments))
        self.queued = []
        return replies


class StandInRedis:
    """A stand-in for a Redis client that keeps strings and sorted sets in memory.

    Attributes:
        values (dict): Each key's value: a string, or a set of members for a sorted set.
        expiries (dict): Each key's expiry in seconds, as last set.
    """

    def __init__(self):
        """Builds an empty stand-in.

        Returns:
            None: This method returns nothing.
        """
        self.values = {}
        self.expiries = {}

    def pipeline(self, transaction=True):
        """Starts a pipeline.

        Args:
            transaction (bool): Ignored; the stand-in always runs commands in order.

        Returns:
            StandInPipeline: A new pipeline.
        """
        return StandInPipeline(self)

    def set(self, key, value):
        """Writes a string.

        Args:
            key (str): The key to write.
            value (str): The value to store.

        Returns:
            bool: Always True.
        """
        self.values[key] = value
        return True

    def expire(self, key, seconds):
        """Records a key's expiry.

        Args:
            key (str): The key to expire.
            seconds (int): How long the key may live.

        Returns:
            bool: Always True.
        """
        self.expiries[key] = seconds
        return True

    def zadd(self, key, mapping):
        """Adds members to a sorted set; every score here is zero, so only the members are kept.

        Args:
            key (str): The sorted set.
            mapping (dict): Each member to its score.

        Returns:
            int: The number of members given.
        """
        members = self.values.setdefault(key, set())
        for member in mapping:
            members.add(member)
        return len(mapping)

    def zrange(self, key, start, stop):
        """Reads a sorted set's members by position, in text order.

        Args:
            key (str): The sorted set.
            start (int): The first position.
            stop (int): The last position, inclusive, or -1 for the end.

        Returns:
            list: The members.
        """
        ordered = sorted(self.values.get(key, set()))
        if stop == -1:
            return ordered[start:]
        return ordered[start:stop + 1]

    def zrangebylex(self, key, minimum, maximum, start=None, num=None):
        """Reads a sorted set's members between an inclusive lower and an exclusive upper text bound.

        Args:
            key (str): The sorted set.
            minimum (bytes): The lower bound, starting with "[".
            maximum (bytes): The upper bound, starting with "(".
            start (int | None): How many matching members to skip.
            num (int | None): The most members to return.

        Returns:
            list: The matching members, in text order.
        """
        lower = minimum[1:]
        upper = maximum[1:]
        matching = []
        for member in sorted(self.values.get(key, set())):
            encoded = member.encode()
            if lower <= encoded < upper:
                matching.append(member)
        first = start or 0
        if num is None:
            return matching[first:]
        return matching[first:first + num]

    def scan_iter(self, match):
        """Lists the keys that start with a pattern's text before its closing "*".

        Args:
            match (str): A pattern ending in "*".

        Returns:
            list: The matching keys, sorted.
        """
        prefix = match.rstrip('*')
        found = []
        for key in sorted(self.values):
            if key.startswith(prefix):
                found.append(key)
        return found

    def delete(self, key):
        """Deletes a key.

        Args:
            key (str): The key to delete.

        Returns:
            int: 1 when the key existed, otherwise 0.
        """
        if key in self.values:
            del self.values[key]
            return 1
        return 0


class SearchingTheCatalogueExample:
    """Writes an option catalogue and searches it through the tier.

    Attributes:
        redis_client (StandInRedis): The stand-in Redis the tier writes into.
        tier (MappingRedisTier): The tier being shown.
        mapping_date (datetime.date): The mapping date of the catalogue.
        segment (str): The segment the options belong to.
        labels (dict): Each written instrument id to a readable description of the contract.
    """

    def __init__(self):
        """Builds the tier around the stand-in.

        Returns:
            None: This method returns nothing.
        """
        self.redis_client = StandInRedis()
        self.tier = MappingRedisTier(MappingRedisConnection(self.redis_client))
        self.mapping_date = datetime.date(2026, 9, 30)
        self.segment = 'nse_equity_index_options'
        self.labels = {}

    def option(self, instrument_identifier, underlying_symbol, expiry_date, strike_price, option_type):
        """Builds one option's identity.

        Args:
            instrument_identifier (str): The instrument id.
            underlying_symbol (str): The underlying's symbol.
            expiry_date (datetime.date): The expiry.
            strike_price (str): The strike, as text.
            option_type (str): "CE" or "PE".

        Returns:
            dict: The identity.
        """
        return {
            'instrument_id': instrument_identifier,
            'exchange': 'nse',
            'segment': self.segment,
            'shape': 'option',
            'symbol': None,
            'underlying_symbol': underlying_symbol,
            'expiry_date': expiry_date,
            'strike_price': decimal.Decimal(strike_price),
            'option_type': option_type,
            'mapping_date': self.mapping_date,
        }

    def write_catalogue(self):
        """Writes six options and their two names into the catalogue and names sets.

        Returns:
            None: This method returns nothing.
        """
        first_expiry = datetime.date(2026, 10, 6)
        second_expiry = datetime.date(2026, 10, 13)
        options = [
            self.option('00000000-0000-4000-8000-000000000001', 'NIFTY', first_expiry, '1000', 'CE'),
            self.option('00000000-0000-4000-8000-000000000002', 'NIFTY', first_expiry, '900', 'PE'),
            self.option('00000000-0000-4000-8000-000000000003', 'NIFTY', first_expiry, '900', 'CE'),
            self.option('00000000-0000-4000-8000-000000000004', 'NIFTY', second_expiry, '25000', 'CE'),
            self.option('00000000-0000-4000-8000-000000000005', 'BANKNIFTY', first_expiry, '55000', 'PE'),
            self.option('00000000-0000-4000-8000-000000000006', 'BANKNIFTY', second_expiry, '55000', 'CE'),
        ]
        catalogue_members = []
        name_members = []
        for identity in options:
            self.labels[identity['instrument_id']] = f'{identity["underlying_symbol"]} {identity["expiry_date"]} {identity["strike_price"]} {identity["option_type"]}'
            catalogue_members.append(self.tier.encode_catalogue_member(identity))
            name = self.tier.catalogue_name(identity)
            if name not in name_members:
                name_members.append(name)
        members_by_key = {
            self.tier.catalogue_key(self.mapping_date, self.segment): catalogue_members,
            self.tier.names_key(self.mapping_date, self.segment): name_members,
        }
        sent = self.tier.write_members(members_by_key, 3600)
        print(f'write_members sent {sent} members')

    def search(self):
        """Reads the catalogue back by position, by prefix, by several prefixes, and by name.

        Returns:
            None: This method returns nothing.
        """
        print('read_catalogue, the first four in order:')
        for instrument_identifier in self.tier.read_catalogue(self.mapping_date, self.segment, 0, 3):
            print(f'  {instrument_identifier} {self.labels[instrument_identifier]}')
        nifty_first_expiry = self.tier.catalogue_prefix('NIFTY', datetime.date(2026, 10, 6))
        print(f'read_catalogue_for_prefix {nifty_first_expiry}:')
        for instrument_identifier in self.tier.read_catalogue_for_prefix(self.mapping_date, self.segment, nifty_first_expiry, 10):
            print(f'  {instrument_identifier} {self.labels[instrument_identifier]}')
        segment_prefixes = [
            (self.segment, self.tier.catalogue_prefix('BANKNIFTY', datetime.date(2026, 10, 13), decimal.Decimal('55000'), 'CE')),
            (self.segment, self.tier.catalogue_prefix('NIFTY', datetime.date(2026, 10, 6), decimal.Decimal('900'), 'CE')),
            (self.segment, self.tier.catalogue_prefix('FINNIFTY')),
        ]
        print(f'read_catalogue_for_prefixes: {self.tier.read_catalogue_for_prefixes(self.mapping_date, segment_prefixes)}')
        print(f'read_names: {self.tier.read_names(self.mapping_date, self.segment)}')

    def clear_yesterday(self):
        """Leaves a key from the day before and clears every date but today's.

        Returns:
            None: This method returns nothing.
        """
        yesterday = self.mapping_date - datetime.timedelta(days=1)
        self.redis_client.set(self.tier.current_date_key(), self.mapping_date.isoformat())
        stale_names = {
            'NIFTY': 0,
        }
        self.redis_client.zadd(self.tier.names_key(yesterday, self.segment), stale_names)
        print(f'Keys before clear_other_dates: {len(self.redis_client.scan_iter(self.tier.KEY_PREFIX + "*"))}')
        print(f'clear_other_dates deleted: {self.tier.clear_other_dates(self.mapping_date)}')
        for key in self.redis_client.scan_iter(self.tier.KEY_PREFIX + '*'):
            print(f'  kept {key}')

    def run(self):
        """Writes the catalogue, searches it, then clears the stale date.

        Returns:
            None: This method returns nothing.
        """
        self.write_catalogue()
        self.search()
        self.clear_yesterday()


if __name__ == '__main__':
    SearchingTheCatalogueExample().run()

It prints:

write_members sent 8 members
read_catalogue, the first four in order:
  00000000-0000-4000-8000-000000000005 BANKNIFTY 2026-10-06 55000 PE
  00000000-0000-4000-8000-000000000006 BANKNIFTY 2026-10-13 55000 CE
  00000000-0000-4000-8000-000000000003 NIFTY 2026-10-06 900 CE
  00000000-0000-4000-8000-000000000002 NIFTY 2026-10-06 900 PE
read_catalogue_for_prefix NIFTY|2026-10-06|:
  00000000-0000-4000-8000-000000000003 NIFTY 2026-10-06 900 CE
  00000000-0000-4000-8000-000000000002 NIFTY 2026-10-06 900 PE
  00000000-0000-4000-8000-000000000001 NIFTY 2026-10-06 1000 CE
read_catalogue_for_prefixes: ['00000000-0000-4000-8000-000000000006', '00000000-0000-4000-8000-000000000003', None]
read_names: ['BANKNIFTY', 'NIFTY']
Keys before clear_other_dates: 4
clear_other_dates deleted: 1
  kept unified:catalogue:2026-09-30:catalogue:nse_equity_index_options
  kept unified:catalogue:2026-09-30:names:nse_equity_index_options
  kept unified:catalogue:current_date
examples/stock_brokers/instruments/mapping/utilities/cache/MappingRedisTier/example_4_when_redis_is_down.py
"""Shows every read and write of the mapping tier degrading quietly when Redis cannot be reached.

The mapping cache sits in front of data that Postgres still holds, so a Redis that is down must never turn an order into an error. Every method of `MappingRedisTier` asks its connection for a client first, and when the answer is None it returns an empty answer: None or an empty dictionary for a read, False or zero for a write. The caller then falls through to Postgres.

This program builds the tier with no client of its own and points `redis_configuration` at port 9 on this machine, where nothing listens, so the first call fails to connect and every later call inside the retry window gets None without trying again. Notice that `failed_connections` stays at one however many calls are made, and that the empty reads keep their usual types, so a caller needs no special case beyond the ones it already has.

Run it from the project root:

    python examples/stock_brokers/instruments/mapping/utilities/cache/MappingRedisTier/example_4_when_redis_is_down.py
"""

import datetime

from stock_brokers.instruments.mapping.utilities.cache import (
    MappingRedisTier,
)
from utilities.configurations import redis_configuration


class WhenRedisIsDownExample:
    """Calls each Redis read and write of the tier while Redis is unreachable.

    Attributes:
        tier (MappingRedisTier): The tier being shown, with a connection it opens itself.
        mapping_date (datetime.date): The mapping date asked about.
        instrument_identifiers (list): The instrument ids asked about.
    """

    def __init__(self):
        """Points the Redis configuration at a closed port and builds the tier.

        Returns:
            None: This method returns nothing.
        """
        redis_configuration['host'] = '127.0.0.1'
        redis_configuration['port'] = 9
        self.tier = MappingRedisTier()
        self.mapping_date = datetime.date(2026, 9, 30)
        self.instrument_identifiers = [
            '0a6e2d41-9c7f-4b58-8d13-6f2e4a9b7c05',
        ]

    def show_reads(self):
        """Prints what each read answers while Redis is down.

        Returns:
            None: This method returns nothing.
        """
        tokens = [
            '408065',
        ]
        segment_prefixes = [
            ('nse_equities', 'INFY|'),
        ]
        print(f'read_current_date: {self.tier.read_current_date()}')
        print(f'read_segment_counts: {self.tier.read_segment_counts(self.mapping_date)}')
        print(f'read_catalogue: {self.tier.read_catalogue(self.mapping_date, "nse_equities", 0, 9)}')
        print(f'read_catalogue_for_prefix: {self.tier.read_catalogue_for_prefix(self.mapping_date, "nse_equities", "INFY|", 5)}')
        print(f'read_catalogue_for_prefixes: {self.tier.read_catalogue_for_prefixes(self.mapping_date, segment_prefixes)}')
        print(f'read_names: {self.tier.read_names(self.mapping_date, "nse_equities")}')
        print(f'read_identities: {self.tier.read_identities(self.mapping_date, self.instrument_identifiers)}')
        print(f'read_token_identifiers: {self.tier.read_token_identifiers(self.mapping_date, "zerodha", tokens)}')
        print(f'read_order_handles: {self.tier.read_order_handles(self.mapping_date, self.instrument_identifiers)}')
        print(f'read_seen: {self.tier.read_seen(self.mapping_date, self.instrument_identifiers)}')
        print(f'read_underlyings: {self.tier.read_underlyings(self.mapping_date, self.instrument_identifiers)}')
        print(f'read_seen_and_underlyings: {self.tier.read_seen_and_underlyings(self.mapping_date, self.instrument_identifiers)}')
        print(f'read_additional_attributes: {self.tier.read_additional_attributes(self.mapping_date, self.instrument_identifiers)}')
        print(f'has_additional_attributes: {self.tier.has_additional_attributes(self.mapping_date)}')

    def show_writes(self):
        """Prints what each write answers while Redis is down.

        Returns:
            None: This method returns nothing.
        """
        identities = {
            self.instrument_identifiers[0]: {
                'instrument_id': self.instrument_identifiers[0],
            },
        }
        handles = {
            self.instrument_identifiers[0]: {
                'zerodha': {
                    'broker_token': '408065',
                    'order_symbol': 'INFY',
                    'lot_size': '1',
                    'tick_size': '0.1',
                },
            },
        }
        fields = {
            self.instrument_identifiers[0]: '2024-01-02|2026-09-30',
        }
        members_by_key = {
            self.tier.names_key(self.mapping_date, 'nse_equities'): [
                'INFY',
            ],
        }
        print(f'write_current_date: {self.tier.write_current_date(self.mapping_date)}')
        print(f'write_identities: {self.tier.write_identities(self.mapping_date, identities)}')
        print(f'write_token_candidates: {self.tier.write_token_candidates(self.mapping_date, "zerodha", {})}')
        print(f'write_order_handles: {self.tier.write_order_handles(self.mapping_date, handles)}')
        print(f'write_fields: {self.tier.write_fields(self.tier.seen_key(self.mapping_date), fields, 60)}')
        print(f'write_members: {self.tier.write_members(members_by_key, 60)}')
        print(f'clear_other_dates: {self.tier.clear_other_dates(self.mapping_date)}')

    def run(self):
        """Shows the reads, then the writes, then how many connection attempts failed.

        Returns:
            None: This method returns nothing.
        """
        self.show_reads()
        self.show_writes()
        print(f'Failed connection attempts: {self.tier.connection.failed_connections}')


if __name__ == '__main__':
    WhenRedisIsDownExample().run()

It prints:

read_current_date: None
read_segment_counts: None
read_catalogue: None
read_catalogue_for_prefix: None
read_catalogue_for_prefixes: None
read_names: None
read_identities: {}
read_token_identifiers: {}
read_order_handles: {}
read_seen: {}
read_underlyings: {}
read_seen_and_underlyings: ({}, {})
read_additional_attributes: {}
has_additional_attributes: False
write_current_date: False
write_identities: False
write_token_candidates: False
write_order_handles: False
write_fields: 0
write_members: 0
clear_other_dates: 0
Failed connection attempts: 1

Methods:

Name Description
current_date_key

The key holding the latest warmed mapping date.
warm_identifier_key

The key holding an identifier that changes on every warm, even one that re-warms the same date.
identity_key

The key of the hash holding one date's identities, keyed by instrument id.
tokens_key

The key of the hash holding one broker's tokens for one date, keyed by broker token.
order_handles_key

The key of the hash holding one date's order handles, keyed by instrument id.
contract_sizes_key

The key of the hash holding one date's contract size decisions, keyed by instrument id.
underlyings_key

The key of the hash holding one date's resolved underlyings: each derivative's instrument id to the instrument id of what it is written on.
additional_attributes_key

The key of the hash holding one date's additional broker attributes, keyed by instrument id.
encode_contract_size

Encode one contract size decision as the JSON the order route reads.
segments_key

The key of the hash holding how many instruments each segment has on one date.
catalogue_key

The key of the sorted set listing one segment's instruments on one date, in name order.
names_key

The key of the sorted set listing one segment's distinct names on one date.
seen_key

The key of the hash holding each instrument's first and last seen dates, keyed by instrument id.
catalogue_name

The name an instrument is listed and searched under: its symbol, or its underlying for a derivative.
encode_catalogue_member

Encode one instrument as its member of a segment's catalogue set.
catalogue_prefix

The start shared by the catalogue members under a name, narrowed by as many identity fields as are given.
decode_catalogue_member

The instrument id at the end of a catalogue member.
read_segment_counts

How many instruments each segment has on a date, which is also the test for a complete catalogue.
read_catalogue

One slice of a segment's catalogue, as instrument ids in name order.
read_catalogue_for_prefix

The instruments whose catalogue members start with a prefix, in expiry, strike and option type order.
read_catalogue_for_prefixes

The first instrument under each of several catalogue prefixes, read in one round trip.
read_names

Every distinct name in a segment's catalogue, in lexical order.
read_seen

Instruments' first and last seen dates.
read_seen_and_underlyings

Instruments' first and last seen dates and their resolved underlyings, read in one round trip.
read_underlyings

Instruments' resolved underlyings.
encode_seen

Encode an instrument's first and last seen dates as the text the seen hash stores.
write_members

Add one batch of members to several lexically ordered sorted sets and set them to expire.
seconds_until_midnight

How long a dated key may live before the calendar day changes under it.
encode_identity

Encode one identity dictionary as the JSON text Redis stores.
decode_identity

Decode one identity dictionary out of the JSON text Redis stores.
encode_order_handles

Encode one instrument's order handles as the JSON text Redis stores.
decode_order_handles

Decode one instrument's order handles out of the JSON text Redis stores.
encode_additional_attributes

Encode one instrument's additional broker attributes as the JSON text Redis stores.
decode_additional_attributes

Decode one instrument's additional broker attributes out of the JSON text Redis stores.
read_additional_attributes

Read instruments' additional broker attributes out of the additional attribute hash.
has_additional_attributes

Say whether the date's additional attribute hash is in Redis at all.
read_current_date

The latest warmed mapping date, as Redis holds it.
write_current_date

Publish the mapping date the hashes now cover, together with a new warm identifier.
read_identities

Read instruments' identities out of the identity hash.
write_identities

Write identities into the identity hash.
read_token_identifiers

Read one broker's tokens out of the token hash, as the instrument ids each carries.
write_token_candidates

Write tokens and the identities they carry, in one pipeline.
read_order_handles

Read instruments' order handles out of the order handle hash.
write_order_handles

Write instruments' order handles into the order handle hash.
write_fields

Write one batch of hash fields and set the hash to expire.
clear_other_dates

Delete every cached mapping date except the one given.
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
156
157
158
159
160
161
162
163
164
165
166
167
def __init__(self, connection=None):
    """
    Build the tier around a Redis connection.

    Args:
        connection (MappingRedisConnection | None): The connection to use. None builds one.

    Returns:
        None: This function returns nothing.
    """
    self.connection = connection or MappingRedisConnection()
    self.KEY_PREFIX = tables.MAPPING_CACHE_PREFIX

current_date_key 

current_date_key()

The key holding the latest warmed mapping date.

Returns:

Type Description
str

The full key.

Example usage

From example_1_keys_and_encodings.py
print(f'current_date_key: {self.tier.current_date_key()}')
From example_3_searching_the_catalogue.py
self.redis_client.set(self.tier.current_date_key(), self.mapping_date.isoformat())
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
169
170
171
172
173
174
175
176
def current_date_key(self):
    """
    The key holding the latest warmed mapping date.

    Returns:
        str: The full key.
    """
    return f"{self.KEY_PREFIX}current_date"

warm_identifier_key 

warm_identifier_key()

The key holding an identifier that changes on every warm, even one that re-warms the same date.

A process that keeps catalogue data in its own memory compares this with the identifier it read the data under, because the mapping date alone does not change when a warm is re-run for the same date.

Returns:

Type Description
str

The full key.

Example usage

From example_1_keys_and_encodings.py
print(f'warm_identifier_key: {self.tier.warm_identifier_key()}')
From example_2_writing_and_reading_one_day.py
print(f'A warm identifier was written: {self.tier.warm_identifier_key() in self.redis_client.values}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
178
179
180
181
182
183
184
185
186
187
def warm_identifier_key(self):
    """
    The key holding an identifier that changes on every warm, even one that re-warms the same date.

    A process that keeps catalogue data in its own memory compares this with the identifier it read the data under, because the mapping date alone does not change when a warm is re-run for the same date.

    Returns:
        str: The full key.
    """
    return f"{self.KEY_PREFIX}warm_identifier"

identity_key 

identity_key(mapping_date)

The key of the hash holding one date's identities, keyed by instrument id.

Parameters:

Name Type Description Default
mapping_date date

The mapping date the hash covers.

 required

Returns:

Type Description
str

The full key.

Example usage

From example_1_keys_and_encodings.py
print(f'identity_key: {self.tier.identity_key(self.mapping_date)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
189
190
191
192
193
194
195
196
197
198
199
def identity_key(self, mapping_date):
    """
    The key of the hash holding one date's identities, keyed by instrument id.

    Args:
        mapping_date (datetime.date): The mapping date the hash covers.

    Returns:
        str: The full key.
    """
    return f"{self.KEY_PREFIX}{mapping_date.isoformat()}:identity"

tokens_key 

tokens_key(mapping_date, broker)

The key of the hash holding one broker's tokens for one date, keyed by broker token.

Parameters:

Name Type Description Default
mapping_date date

The mapping date the hash covers.

 required
broker str

The broker name, for example "zerodha".

 required

Returns:

Type Description
str

The full key.

Example usage

From example_1_keys_and_encodings.py
print(f'tokens_key: {self.tier.tokens_key(self.mapping_date, "zerodha")}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
201
202
203
204
205
206
207
208
209
210
211
212
def tokens_key(self, mapping_date, broker):
    """
    The key of the hash holding one broker's tokens for one date, keyed by broker token.

    Args:
        mapping_date (datetime.date): The mapping date the hash covers.
        broker (str): The broker name, for example "zerodha".

    Returns:
        str: The full key.
    """
    return f"{self.KEY_PREFIX}{mapping_date.isoformat()}:tokens:{broker}"

order_handles_key 

order_handles_key(mapping_date)

The key of the hash holding one date's order handles, keyed by instrument id.

The key says "order_handles" rather than "brokers" because the value stored under it widened from a bare token to the whole handle an order needs. A key of its own means a Redis still holding the narrower value is a miss rather than a decode that succeeds into the wrong shape.

Parameters:

Name Type Description Default
mapping_date date

The mapping date the hash covers.

 required

Returns:

Type Description
str

The full key.

Example usage

From example_1_keys_and_encodings.py
print(f'order_handles_key: {self.tier.order_handles_key(self.mapping_date)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
214
215
216
217
218
219
220
221
222
223
224
225
226
def order_handles_key(self, mapping_date):
    """
    The key of the hash holding one date's order handles, keyed by instrument id.

    The key says "order_handles" rather than "brokers" because the value stored under it widened from a bare token to the whole handle an order needs. A key of its own means a Redis still holding the narrower value is a miss rather than a decode that succeeds into the wrong shape.

    Args:
        mapping_date (datetime.date): The mapping date the hash covers.

    Returns:
        str: The full key.
    """
    return f"{self.KEY_PREFIX}{mapping_date.isoformat()}:order_handles"

contract_sizes_key 

contract_sizes_key(mapping_date)

The key of the hash holding one date's contract size decisions, keyed by instrument id.

Parameters:

Name Type Description Default
mapping_date date

The mapping date the hash covers.

 required

Returns:

Type Description
str

The full key.

Example usage

From example_1_keys_and_encodings.py
print(f'contract_sizes_key: {self.tier.contract_sizes_key(self.mapping_date)}')
From example_2_writing_and_reading_one_day.py
written += self.tier.write_fields(self.tier.contract_sizes_key(self.mapping_date), contract_size_fields, expiry_seconds)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
228
229
230
231
232
233
234
235
236
237
238
def contract_sizes_key(self, mapping_date):
    """
    The key of the hash holding one date's contract size decisions, keyed by instrument id.

    Args:
        mapping_date (datetime.date): The mapping date the hash covers.

    Returns:
        str: The full key.
    """
    return f"{self.KEY_PREFIX}{mapping_date.isoformat()}:contract_sizes"

underlyings_key 

underlyings_key(mapping_date)

The key of the hash holding one date's resolved underlyings: each derivative's instrument id to the instrument id of what it is written on.

Parameters:

Name Type Description Default
mapping_date date

The mapping date the hash covers.

 required

Returns:

Type Description
str

The full key.

Example usage

From example_1_keys_and_encodings.py
print(f'underlyings_key: {self.tier.underlyings_key(self.mapping_date)}')
From example_2_writing_and_reading_one_day.py
written += self.tier.write_fields(self.tier.underlyings_key(self.mapping_date), underlying_fields, expiry_seconds)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
240
241
242
243
244
245
246
247
248
249
250
def underlyings_key(self, mapping_date):
    """
    The key of the hash holding one date's resolved underlyings: each derivative's instrument id to the instrument id of what it is written on.

    Args:
        mapping_date (datetime.date): The mapping date the hash covers.

    Returns:
        str: The full key.
    """
    return f"{self.KEY_PREFIX}{mapping_date.isoformat()}:underlyings"

additional_attributes_key 

additional_attributes_key(mapping_date)

The key of the hash holding one date's additional broker attributes, keyed by instrument id.

This is a hash of its own rather than more fields on the order handles, because an order needs the handle on every request and nothing needs these attributes to place one. Keeping them apart is what lets the order path stay at the 521 bytes per instrument it costs today.

Parameters:

Name Type Description Default
mapping_date date

The mapping date the hash covers.

 required

Returns:

Type Description
str

The full key.

Example usage

From example_1_keys_and_encodings.py
print(f'additional_attributes_key: {self.tier.additional_attributes_key(self.mapping_date)}')
From example_2_writing_and_reading_one_day.py
written += self.tier.write_fields(self.tier.additional_attributes_key(self.mapping_date), attribute_fields, expiry_seconds)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
252
253
254
255
256
257
258
259
260
261
262
263
264
def additional_attributes_key(self, mapping_date):
    """
    The key of the hash holding one date's additional broker attributes, keyed by instrument id.

    This is a hash of its own rather than more fields on the order handles, because an order needs the handle on every request and nothing needs these attributes to place one. Keeping them apart is what lets the order path stay at the 521 bytes per instrument it costs today.

    Args:
        mapping_date (datetime.date): The mapping date the hash covers.

    Returns:
        str: The full key.
    """
    return f"{self.KEY_PREFIX}{mapping_date.isoformat()}:additional_attributes"

encode_contract_size 

encode_contract_size(units_per_lot, status, tradeable)

Encode one contract size decision as the JSON the order route reads.

The size is kept as text, as lot sizes are in order handles, so it survives the round trip exactly.

Parameters:

Name Type Description Default
units_per_lot Decimal | None

Quotation units per lot, or None when undecided.

 required
status str

The decision's status.

 required
tradeable bool

Whether orders may be sent.

 required

Returns:

Type Description
str

The JSON text.

Example usage

From example_1_keys_and_encodings.py
size = self.tier.encode_contract_size(decimal.Decimal('1000.000'), 'confirmed', True)
From example_1_keys_and_encodings.py
no_source = self.tier.encode_contract_size(None, 'no_source', False)
From example_2_writing_and_reading_one_day.py
contract_size_fields = {
    self.FUTURE_ID: self.tier.encode_contract_size(decimal.Decimal('75'), 'confirmed', True),
}
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
def encode_contract_size(self, units_per_lot, status, tradeable):
    """
    Encode one contract size decision as the JSON the order route reads.

    The size is kept as text, as lot sizes are in order handles, so it survives the round trip exactly.

    Args:
        units_per_lot (decimal.Decimal | None): Quotation units per lot, or None when undecided.
        status (str): The decision's status.
        tradeable (bool): Whether orders may be sent.

    Returns:
        str: The JSON text.
    """
    size_text = None
    if units_per_lot is not None:
        size_text = format(decimal.Decimal(units_per_lot).normalize(), "f")
    return json.dumps({
        "units_per_lot": size_text,
        "status": status,
        "tradeable": bool(tradeable),
    })

segments_key 

segments_key(mapping_date)

The key of the hash holding how many instruments each segment has on one date.

It is also what says a date's catalogue is complete: the warm writes it after every catalogue and names set, so a reader that finds it can trust the rest.

Parameters:

Name Type Description Default
mapping_date date

The mapping date the hash covers.

 required

Returns:

Type Description
str

The full key.

Example usage

From example_1_keys_and_encodings.py
print(f'segments_key: {self.tier.segments_key(self.mapping_date)}')
From example_2_writing_and_reading_one_day.py
written += self.tier.write_fields(self.tier.segments_key(self.mapping_date), segment_fields, expiry_seconds)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
289
290
291
292
293
294
295
296
297
298
299
300
301
def segments_key(self, mapping_date):
    """
    The key of the hash holding how many instruments each segment has on one date.

    It is also what says a date's catalogue is complete: the warm writes it after every catalogue and names set, so a reader that finds it can trust the rest.

    Args:
        mapping_date (datetime.date): The mapping date the hash covers.

    Returns:
        str: The full key.
    """
    return f"{self.KEY_PREFIX}{mapping_date.isoformat()}:segments"

catalogue_key 

catalogue_key(mapping_date, segment)

The key of the sorted set listing one segment's instruments on one date, in name order.

Parameters:

Name Type Description Default
mapping_date date

The mapping date the set covers.

 required
segment str

The exchange-prefixed segment, for example "nse_equities".

 required

Returns:

Type Description
str

The full key.

Example usage

From example_1_keys_and_encodings.py
print(f'catalogue_key: {self.tier.catalogue_key(self.mapping_date, "nse_equities")}')
From example_3_searching_the_catalogue.py
members_by_key = {
    self.tier.catalogue_key(self.mapping_date, self.segment): catalogue_members,
    self.tier.names_key(self.mapping_date, self.segment): name_members,
}
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
303
304
305
306
307
308
309
310
311
312
313
314
def catalogue_key(self, mapping_date, segment):
    """
    The key of the sorted set listing one segment's instruments on one date, in name order.

    Args:
        mapping_date (datetime.date): The mapping date the set covers.
        segment (str): The exchange-prefixed segment, for example "nse_equities".

    Returns:
        str: The full key.
    """
    return f"{self.KEY_PREFIX}{mapping_date.isoformat()}:catalogue:{segment}"

names_key 

names_key(mapping_date, segment)

The key of the sorted set listing one segment's distinct names on one date.

A name is a security's symbol or a derivative's underlying. An option segment has hundreds of thousands of contracts but only a few hundred underlyings, so a search scans this set rather than the catalogue.

Parameters:

Name Type Description Default
mapping_date date

The mapping date the set covers.

 required
segment str

The exchange-prefixed segment, for example "nse_equity_options".

 required

Returns:

Type Description
str

The full key.

Example usage

From example_1_keys_and_encodings.py
print(f'names_key: {self.tier.names_key(self.mapping_date, "nse_equity_index_options")}')
From example_3_searching_the_catalogue.py
members_by_key = {
    self.tier.catalogue_key(self.mapping_date, self.segment): catalogue_members,
    self.tier.names_key(self.mapping_date, self.segment): name_members,
}
From example_3_searching_the_catalogue.py
self.redis_client.zadd(self.tier.names_key(yesterday, self.segment), stale_names)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
316
317
318
319
320
321
322
323
324
325
326
327
328
329
def names_key(self, mapping_date, segment):
    """
    The key of the sorted set listing one segment's distinct names on one date.

    A name is a security's symbol or a derivative's underlying. An option segment has hundreds of thousands of contracts but only a few hundred underlyings, so a search scans this set rather than the catalogue.

    Args:
        mapping_date (datetime.date): The mapping date the set covers.
        segment (str): The exchange-prefixed segment, for example "nse_equity_options".

    Returns:
        str: The full key.
    """
    return f"{self.KEY_PREFIX}{mapping_date.isoformat()}:names:{segment}"

seen_key 

seen_key(mapping_date)

The key of the hash holding each instrument's first and last seen dates, keyed by instrument id.

Parameters:

Name Type Description Default
mapping_date date

The mapping date the hash covers.

 required

Returns:

Type Description
str

The full key.

Example usage

From example_1_keys_and_encodings.py
print(f'seen_key: {self.tier.seen_key(self.mapping_date)}')
From example_2_writing_and_reading_one_day.py
written += self.tier.write_fields(self.tier.seen_key(self.mapping_date), seen_fields, expiry_seconds)
From example_2_writing_and_reading_one_day.py
print(f'Expiry set on the seen hash: {self.redis_client.expiries[self.tier.seen_key(self.mapping_date)]} seconds')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
331
332
333
334
335
336
337
338
339
340
341
def seen_key(self, mapping_date):
    """
    The key of the hash holding each instrument's first and last seen dates, keyed by instrument id.

    Args:
        mapping_date (datetime.date): The mapping date the hash covers.

    Returns:
        str: The full key.
    """
    return f"{self.KEY_PREFIX}{mapping_date.isoformat()}:seen"

catalogue_name 

catalogue_name(identity)

The name an instrument is listed and searched under: its symbol, or its underlying for a derivative.

Parameters:

Name Type Description Default
identity dict

The instrument's identity.

 required

Returns:

Type Description
str

The name, upper-cased, with any "|" replaced so it cannot break the member layout.

Example usage

From example_1_keys_and_encodings.py
print(f'Catalogue name of the equity: {self.tier.catalogue_name(self.equity_identity)}')
From example_1_keys_and_encodings.py
print(f'Catalogue name of the option: {self.tier.catalogue_name(self.option_identity)}')
From example_3_searching_the_catalogue.py
name = self.tier.catalogue_name(identity)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
343
344
345
346
347
348
349
350
351
352
353
354
def catalogue_name(self, identity):
    """
    The name an instrument is listed and searched under: its symbol, or its underlying for a derivative.

    Args:
        identity (dict): The instrument's identity.

    Returns:
        str: The name, upper-cased, with any "|" replaced so it cannot break the member layout.
    """
    name = identity["symbol"] if identity["shape"] == "security" else identity["underlying_symbol"]
    return (name or "").upper().replace("|", "/")

encode_catalogue_member 

encode_catalogue_member(identity)

Encode one instrument as its member of a segment's catalogue set.

Every member has the score zero, so the set orders lexically, and the member is laid out as NAME|expiry|strike|option_type|instrument_id so that lexical order is name, then expiry, then strike, then option type. The strike is zero-padded to a fixed width for the same reason: "00000000900.0000" sorts before "00000001000.0000" where "900" would sort after "1000".

Parameters:

Name Type Description Default
identity dict

The instrument's identity.

 required

Returns:

Type Description
str

The member.

Example usage

From example_1_keys_and_encodings.py
equity_member = self.tier.encode_catalogue_member(self.equity_identity)
From example_1_keys_and_encodings.py
option_member = self.tier.encode_catalogue_member(self.option_identity)
From example_3_searching_the_catalogue.py
catalogue_members.append(self.tier.encode_catalogue_member(identity))
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
def encode_catalogue_member(self, identity):
    """
    Encode one instrument as its member of a segment's catalogue set.

    Every member has the score zero, so the set orders lexically, and the member is laid out as ``NAME|expiry|strike|option_type|instrument_id`` so that lexical order is name, then expiry, then strike, then option type. The strike is zero-padded to a fixed width for the same reason: ``"00000000900.0000"`` sorts before ``"00000001000.0000"`` where ``"900"`` would sort after ``"1000"``.

    Args:
        identity (dict): The instrument's identity.

    Returns:
        str: The member.
    """
    expiry = identity["expiry_date"]
    strike = identity["strike_price"]
    return "|".join((
        self.catalogue_name(identity),
        expiry.isoformat() if expiry else "",
        self._catalogue_strike(strike) if strike is not None else "",
        identity["option_type"] or "",
        identity["instrument_id"],
    ))

_catalogue_strike 

_catalogue_strike(strike_price)

A strike as the catalogue spells it: zero-padded to a fixed width so lexical order is numeric order.

Parameters:

Name Type Description Default
strike_price Decimal | int | str

The strike.

 required

Returns:

Type Description
str

The strike, for example "00000025000.0000".
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
378
379
380
381
382
383
384
385
386
387
388
def _catalogue_strike(self, strike_price):
    """
    A strike as the catalogue spells it: zero-padded to a fixed width so lexical order is numeric order.

    Args:
        strike_price (decimal.Decimal | int | str): The strike.

    Returns:
        str: The strike, for example "00000025000.0000".
    """
    return f"{decimal.Decimal(strike_price):016.4f}"

catalogue_prefix 

catalogue_prefix(name, expiry_date=None, strike_price=None, option_type=None)

The start shared by the catalogue members under a name, narrowed by as many identity fields as are given.

The prefix stops after the first field not given, each part ending in "|": a name alone gives NIFTY|, which every contract under NIFTY shares and no contract under NIFTYNXT50 does; a name and an expiry give the futures and options of that expiry; a whole option identity gives exactly one member. A security's member has nothing after its name, so its name alone finds it.

Parameters:

Name Type Description Default
name str

The upper-cased name.

 required
expiry_date date | None

The expiry, for a derivative.

 None
strike_price Decimal | None

The strike, for an option.

 None
option_type str | None

"CE" or "PE", for an option.

 None

Returns:

Type Description
str

The prefix, ending in "|".

Example usage

From example_1_keys_and_encodings.py
expiry_prefix = self.tier.catalogue_prefix('NIFTY', datetime.date(2026, 10, 6))
From example_1_keys_and_encodings.py
whole_prefix = self.tier.catalogue_prefix(
    'NIFTY',
    datetime.date(2026, 10, 6),
    decimal.Decimal('25000'),
    'CE',
)
From example_1_keys_and_encodings.py
print(f'Prefix for a name alone: {self.tier.catalogue_prefix("NIFTY")}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
def catalogue_prefix(self, name, expiry_date=None, strike_price=None, option_type=None):
    """
    The start shared by the catalogue members under a name, narrowed by as many identity fields as are given.

    The prefix stops after the first field not given, each part ending in "|": a name alone gives ``NIFTY|``, which every contract under NIFTY shares and no contract under NIFTYNXT50 does; a name and an expiry give the futures and options of that expiry; a whole option identity gives exactly one member. A security's member has nothing after its name, so its name alone finds it.

    Args:
        name (str): The upper-cased name.
        expiry_date (datetime.date | None): The expiry, for a derivative.
        strike_price (decimal.Decimal | None): The strike, for an option.
        option_type (str | None): "CE" or "PE", for an option.

    Returns:
        str: The prefix, ending in "|".
    """
    parts = [name]
    for value in (expiry_date.isoformat() if expiry_date else None,
                  self._catalogue_strike(strike_price) if strike_price is not None else None,
                  option_type or None):
        if value is None:
            break
        parts.append(value)
    return "|".join(parts) + "|"

decode_catalogue_member 

decode_catalogue_member(member)

The instrument id at the end of a catalogue member.

Parameters:

Name Type Description Default
member bytes | str

The member as Redis returned it.

 required

Returns:

Type Description
str

The instrument id.

Example usage

From example_1_keys_and_encodings.py
print(f'Instrument id read back from the option member: {self.tier.decode_catalogue_member(option_member)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
414
415
416
417
418
419
420
421
422
423
424
def decode_catalogue_member(self, member):
    """
    The instrument id at the end of a catalogue member.

    Args:
        member (bytes | str): The member as Redis returned it.

    Returns:
        str: The instrument id.
    """
    return as_text(member).rsplit("|", 1)[1]

read_segment_counts 

read_segment_counts(mapping_date)

How many instruments each segment has on a date, which is also the test for a complete catalogue.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required

Returns:

Type Description
dict | None

Mapping of segment to instrument count, or None when Redis is unreachable or the date's catalogue has not been warmed.

Example usage

From example_2_writing_and_reading_one_day.py
print(f'read_segment_counts: {self.tier.read_segment_counts(self.mapping_date)}')
From example_2_writing_and_reading_one_day.py
print(f'read_segment_counts yesterday: {self.tier.read_segment_counts(yesterday)}')
From example_4_when_redis_is_down.py
print(f'read_segment_counts: {self.tier.read_segment_counts(self.mapping_date)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
def read_segment_counts(self, mapping_date):
    """
    How many instruments each segment has on a date, which is also the test for a complete catalogue.

    Args:
        mapping_date (datetime.date): The mapping date to read.

    Returns:
        dict | None: Mapping of segment to instrument count, or None when Redis is unreachable or the date's catalogue has not been warmed.
    """
    client = self.connection.client()
    if client is None:
        return None
    try:
        stored = client.hgetall(self.segments_key(mapping_date))
    except redis.RedisError:
        return None
    if not stored:
        return None
    counts = {}
    for segment, count in stored.items():
        counts[as_text(segment)] = int(as_text(count))
    return counts

read_catalogue 

read_catalogue(mapping_date, segment, start, stop)

One slice of a segment's catalogue, as instrument ids in name order.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required
segment str

The exchange-prefixed segment.

 required
start int

The first position to read.

 required
stop int

The last position to read, inclusive.

 required

Returns:

Type Description
list[str] | None

The instrument ids, or None when Redis is not usable.

Example usage

From example_3_searching_the_catalogue.py
self.tier.read_catalogue(self.mapping_date, self.segment, 0, 3)
From example_4_when_redis_is_down.py
print(f'read_catalogue: {self.tier.read_catalogue(self.mapping_date, "nse_equities", 0, 9)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
def read_catalogue(self, mapping_date, segment, start, stop):
    """
    One slice of a segment's catalogue, as instrument ids in name order.

    Args:
        mapping_date (datetime.date): The mapping date to read.
        segment (str): The exchange-prefixed segment.
        start (int): The first position to read.
        stop (int): The last position to read, inclusive.

    Returns:
        list[str] | None: The instrument ids, or None when Redis is not usable.
    """
    client = self.connection.client()
    if client is None:
        return None
    try:
        members = client.zrange(self.catalogue_key(mapping_date, segment), start, stop)
    except redis.RedisError:
        return None
    return [self.decode_catalogue_member(member) for member in members]

read_catalogue_for_prefix 

read_catalogue_for_prefix(mapping_date, segment, prefix, limit)

The instruments whose catalogue members start with a prefix, in expiry, strike and option type order.

A prefix from catalogue_prefix with only a name lists every contract under that name; with a whole identity it finds that one instrument.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required
segment str

The exchange-prefixed segment.

 required
prefix str

The member prefix, ending in "|".

 required
limit int

The most instrument ids to return.

 required

Returns:

Type Description
list[str] | None

The instrument ids, or None when Redis is not usable.

Example usage

From example_3_searching_the_catalogue.py
self.tier.read_catalogue_for_prefix(self.mapping_date, self.segment, nifty_first_expiry, 10)
From example_4_when_redis_is_down.py
print(f'read_catalogue_for_prefix: {self.tier.read_catalogue_for_prefix(self.mapping_date, "nse_equities", "INFY|", 5)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
def read_catalogue_for_prefix(self, mapping_date, segment, prefix, limit):
    """
    The instruments whose catalogue members start with a prefix, in expiry, strike and option type order.

    A prefix from catalogue_prefix with only a name lists every contract under that name; with a whole identity it finds that one instrument.

    Args:
        mapping_date (datetime.date): The mapping date to read.
        segment (str): The exchange-prefixed segment.
        prefix (str): The member prefix, ending in "|".
        limit (int): The most instrument ids to return.

    Returns:
        list[str] | None: The instrument ids, or None when Redis is not usable.
    """
    client = self.connection.client()
    if client is None:
        return None
    prefix = prefix.encode()
    try:
        members = client.zrangebylex(self.catalogue_key(mapping_date, segment),
                                     b"[" + prefix, b"(" + prefix + b"\xff", start=0, num=limit)
    except redis.RedisError:
        return None
    return [self.decode_catalogue_member(member) for member in members]

read_catalogue_for_prefixes 

read_catalogue_for_prefixes(mapping_date, segment_prefixes)

The first instrument under each of several catalogue prefixes, read in one round trip.

Each prefix is looked up as read_catalogue_for_prefix looks up one with a limit of one, but the look-ups share one pipeline, so finding many instruments by their identity fields costs one round trip rather than one each.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required
segment_prefixes list[tuple[str, str]]

Pairs of an exchange-prefixed segment and a member prefix from catalogue_prefix, the prefix ending in "|".

 required

Returns:

Type Description
list[str | None] | None

The first instrument id under each prefix, or None where no member starts with it, in the order asked; None when Redis is not usable.

Example usage

From example_3_searching_the_catalogue.py
print(f'read_catalogue_for_prefixes: {self.tier.read_catalogue_for_prefixes(self.mapping_date, segment_prefixes)}')
From example_4_when_redis_is_down.py
print(f'read_catalogue_for_prefixes: {self.tier.read_catalogue_for_prefixes(self.mapping_date, segment_prefixes)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
def read_catalogue_for_prefixes(self, mapping_date, segment_prefixes):
    """
    The first instrument under each of several catalogue prefixes, read in one round trip.

    Each prefix is looked up as read_catalogue_for_prefix looks up one with a limit of one, but the look-ups share one pipeline, so finding many instruments by their identity fields costs one round trip rather than one each.

    Args:
        mapping_date (datetime.date): The mapping date to read.
        segment_prefixes (list[tuple[str, str]]): Pairs of an exchange-prefixed segment and a member prefix from catalogue_prefix, the prefix ending in "|".

    Returns:
        list[str | None] | None: The first instrument id under each prefix, or None where no member starts with it, in the order asked; None when Redis is not usable.
    """
    client = self.connection.client()
    if client is None:
        return None
    if not segment_prefixes:
        return []
    pipeline = client.pipeline(transaction=False)
    for segment, prefix in segment_prefixes:
        encoded_prefix = prefix.encode()
        pipeline.zrangebylex(self.catalogue_key(mapping_date, segment),
                             b"[" + encoded_prefix, b"(" + encoded_prefix + b"\xff", start=0, num=1)
    try:
        replies = pipeline.execute()
    except redis.RedisError:
        return None
    identifiers = []
    for members in replies:
        if members:
            identifiers.append(self.decode_catalogue_member(members[0]))
        else:
            identifiers.append(None)
    return identifiers

read_names 

read_names(mapping_date, segment)

Every distinct name in a segment's catalogue, in lexical order.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required
segment str

The exchange-prefixed segment.

 required

Returns:

Type Description
list[str] | None

The names, or None when Redis is not usable.

Example usage

From example_3_searching_the_catalogue.py
print(f'read_names: {self.tier.read_names(self.mapping_date, self.segment)}')
From example_4_when_redis_is_down.py
print(f'read_names: {self.tier.read_names(self.mapping_date, "nse_equities")}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
def read_names(self, mapping_date, segment):
    """
    Every distinct name in a segment's catalogue, in lexical order.

    Args:
        mapping_date (datetime.date): The mapping date to read.
        segment (str): The exchange-prefixed segment.

    Returns:
        list[str] | None: The names, or None when Redis is not usable.
    """
    client = self.connection.client()
    if client is None:
        return None
    try:
        names = client.zrange(self.names_key(mapping_date, segment), 0, -1)
    except redis.RedisError:
        return None
    return [as_text(name) for name in names]

read_seen 

read_seen(mapping_date, instrument_identifiers)

Instruments' first and last seen dates.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required
instrument_identifiers list[str]

The instrument ids to read.

 required

Returns:

Type Description
dict

Mapping of instrument id to a (first_seen_date, last_seen_date) pair of dates. Instruments Redis did not hold are absent.

Example usage

From example_2_writing_and_reading_one_day.py
print(f'read_seen: {self.tier.read_seen(self.mapping_date, both)}')
From example_4_when_redis_is_down.py
print(f'read_seen: {self.tier.read_seen(self.mapping_date, self.instrument_identifiers)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
def read_seen(self, mapping_date, instrument_identifiers):
    """
    Instruments' first and last seen dates.

    Args:
        mapping_date (datetime.date): The mapping date to read.
        instrument_identifiers (list[str]): The instrument ids to read.

    Returns:
        dict: Mapping of instrument id to a (first_seen_date, last_seen_date) pair of dates. Instruments Redis did not hold are absent.
    """
    client = self.connection.client()
    if client is None or not instrument_identifiers:
        return {}
    try:
        stored = client.hmget(self.seen_key(mapping_date), instrument_identifiers)
    except redis.RedisError:
        return {}
    seen = {}
    for position, instrument_identifier in enumerate(instrument_identifiers):
        value = as_text(stored[position])
        if value:
            first, last = value.split("|")
            seen[instrument_identifier] = (datetime.date.fromisoformat(first), datetime.date.fromisoformat(last))
    return seen

read_seen_and_underlyings 

read_seen_and_underlyings(mapping_date, instrument_identifiers)

Instruments' first and last seen dates and their resolved underlyings, read in one round trip.

The two hashes are read together because /details needs both for the same instruments, and a separate read would add a round trip to every details request.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required
instrument_identifiers list[str]

The instrument ids to read.

 required

Returns:

Type Description
tuple

A dict of instrument id to a (first_seen_date, last_seen_date) pair of dates, and a dict of instrument id to its underlying's instrument id. Instruments Redis did not hold, and instruments with no resolved underlying, are absent.

Example usage

From example_2_writing_and_reading_one_day.py
seen, underlyings = self.tier.read_seen_and_underlyings(self.mapping_date, both)
From example_4_when_redis_is_down.py
print(f'read_seen_and_underlyings: {self.tier.read_seen_and_underlyings(self.mapping_date, self.instrument_identifiers)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
def read_seen_and_underlyings(self, mapping_date, instrument_identifiers):
    """
    Instruments' first and last seen dates and their resolved underlyings, read in one round trip.

    The two hashes are read together because `/details` needs both for the same instruments, and a separate read would add a round trip to every details request.

    Args:
        mapping_date (datetime.date): The mapping date to read.
        instrument_identifiers (list[str]): The instrument ids to read.

    Returns:
        tuple: A dict of instrument id to a (first_seen_date, last_seen_date) pair of dates, and a dict of instrument id to its underlying's instrument id. Instruments Redis did not hold, and instruments with no resolved underlying, are absent.
    """
    client = self.connection.client()
    if client is None or not instrument_identifiers:
        return {}, {}
    try:
        pipeline = client.pipeline(transaction=False)
        pipeline.hmget(self.seen_key(mapping_date), instrument_identifiers)
        pipeline.hmget(self.underlyings_key(mapping_date), instrument_identifiers)
        stored_seen, stored_underlyings = pipeline.execute()
    except redis.RedisError:
        return {}, {}
    seen = {}
    underlyings = {}
    for position, instrument_identifier in enumerate(instrument_identifiers):
        value = as_text(stored_seen[position])
        if value:
            first, last = value.split("|")
            seen[instrument_identifier] = (datetime.date.fromisoformat(first), datetime.date.fromisoformat(last))
        underlying = as_text(stored_underlyings[position])
        if underlying:
            underlyings[instrument_identifier] = underlying
    return seen, underlyings

read_underlyings 

read_underlyings(mapping_date, instrument_identifiers)

Instruments' resolved underlyings.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required
instrument_identifiers list[str]

The instrument ids to read.

 required

Returns:

Type Description
dict

Mapping of instrument id to its underlying's instrument id. Instruments with no resolved underlying are absent.

Example usage

From example_2_writing_and_reading_one_day.py
print(f'read_underlyings: {self.tier.read_underlyings(self.mapping_date, both)}')
From example_4_when_redis_is_down.py
print(f'read_underlyings: {self.tier.read_underlyings(self.mapping_date, self.instrument_identifiers)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
def read_underlyings(self, mapping_date, instrument_identifiers):
    """
    Instruments' resolved underlyings.

    Args:
        mapping_date (datetime.date): The mapping date to read.
        instrument_identifiers (list[str]): The instrument ids to read.

    Returns:
        dict: Mapping of instrument id to its underlying's instrument id. Instruments with no resolved underlying are absent.
    """
    client = self.connection.client()
    if client is None or not instrument_identifiers:
        return {}
    try:
        stored = client.hmget(self.underlyings_key(mapping_date), instrument_identifiers)
    except redis.RedisError:
        return {}
    underlyings = {}
    for position, instrument_identifier in enumerate(instrument_identifiers):
        underlying = as_text(stored[position])
        if underlying:
            underlyings[instrument_identifier] = underlying
    return underlyings

encode_seen 

encode_seen(first_seen_date, last_seen_date)

Encode an instrument's first and last seen dates as the text the seen hash stores.

Parameters:

Name Type Description Default
first_seen_date date

The first date the instrument was mapped.

 required
last_seen_date date

The last date the instrument was mapped.

 required

Returns:

Type Description
str

The two dates, YYYY-MM-DD|YYYY-MM-DD.

Example usage

From example_1_keys_and_encodings.py
seen = self.tier.encode_seen(datetime.date(2024, 1, 2), self.mapping_date)
From example_2_writing_and_reading_one_day.py
seen_fields = {
    self.FUTURE_ID: self.tier.encode_seen(datetime.date(2026, 7, 1), self.mapping_date),
    self.INDEX_ID: self.tier.encode_seen(datetime.date(2024, 1, 2), self.mapping_date),
}
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
639
640
641
642
643
644
645
646
647
648
649
650
def encode_seen(self, first_seen_date, last_seen_date):
    """
    Encode an instrument's first and last seen dates as the text the seen hash stores.

    Args:
        first_seen_date (datetime.date): The first date the instrument was mapped.
        last_seen_date (datetime.date): The last date the instrument was mapped.

    Returns:
        str: The two dates, ``YYYY-MM-DD|YYYY-MM-DD``.
    """
    return f"{first_seen_date.isoformat()}|{last_seen_date.isoformat()}"

write_members 

write_members(members_by_key, expiry_seconds)

Add one batch of members to several lexically ordered sorted sets and set them to expire.

Parameters:

Name Type Description Default
members_by_key dict

Mapping of sorted set key to the members to add to it, all with score zero.

 required
expiry_seconds int

How long each set may live.

 required

Returns:

Type Description
int

The number of members sent, which is zero when there were none or when Redis was not usable.

Example usage

From example_3_searching_the_catalogue.py
sent = self.tier.write_members(members_by_key, 3600)
From example_4_when_redis_is_down.py
print(f'write_members: {self.tier.write_members(members_by_key, 60)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
def write_members(self, members_by_key, expiry_seconds):
    """
    Add one batch of members to several lexically ordered sorted sets and set them to expire.

    Args:
        members_by_key (dict): Mapping of sorted set key to the members to add to it, all with score zero.
        expiry_seconds (int): How long each set may live.

    Returns:
        int: The number of members sent, which is zero when there were none or when Redis was not usable.
    """
    client = self.connection.client()
    if client is None:
        return 0
    sent = 0
    try:
        pipeline = client.pipeline()
        for key, members in members_by_key.items():
            if not members:
                continue
            pipeline.zadd(key, dict.fromkeys(members, 0))
            pipeline.expire(key, expiry_seconds)
            sent += len(members)
        pipeline.execute()
    except redis.RedisError:
        return 0
    return sent

seconds_until_midnight 

seconds_until_midnight()

How long a dated key may live before the calendar day changes under it.

A mapping is superseded the next morning rather than at midnight, so this is an upper bound rather than a true expiry. It exists so that a Redis nobody tends cannot accumulate a month of stale mappings; the mapping date in every key is what actually enforces freshness.

Returns:

Type Description
int

Seconds remaining until midnight, never less than sixty.

Example usage

From example_1_keys_and_encodings.py
print(f'Keys live at least a minute: {self.tier.seconds_until_midnight() >= 60}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
680
681
682
683
684
685
686
687
688
689
690
691
def seconds_until_midnight(self):
    """
    How long a dated key may live before the calendar day changes under it.

    A mapping is superseded the next morning rather than at midnight, so this is an upper bound rather than a true expiry. It exists so that a Redis nobody tends cannot accumulate a month of stale mappings; the mapping date in every key is what actually enforces freshness.

    Returns:
        int: Seconds remaining until midnight, never less than sixty.
    """
    now = datetime.datetime.now()
    midnight = datetime.datetime.combine(now.date() + datetime.timedelta(days=1), datetime.time.min)
    return max(60, int((midnight - now).total_seconds()))

encode_identity 

encode_identity(identity)

Encode one identity dictionary as the JSON text Redis stores.

Parameters:

Name Type Description Default
identity dict

The ten identity fields a resolution returns.

 required

Returns:

Type Description
str

The identity as JSON, with dates as ISO text and the strike price as a decimal string.

Example usage

From example_1_keys_and_encodings.py
encoded_identity = self.tier.encode_identity(self.option_identity)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
def encode_identity(self, identity):
    """
    Encode one identity dictionary as the JSON text Redis stores.

    Args:
        identity (dict): The ten identity fields a resolution returns.

    Returns:
        str: The identity as JSON, with dates as ISO text and the strike price as a decimal string.
    """
    encodable = {}
    for field_name, value in identity.items():
        if value is None:
            encodable[field_name] = None
        elif isinstance(value, datetime.date):
            encodable[field_name] = value.isoformat()
        else:
            encodable[field_name] = str(value)
    return json.dumps(encodable)

decode_identity 

decode_identity(encoded)

Decode one identity dictionary out of the JSON text Redis stores.

Parameters:

Name Type Description Default
encoded bytes | str

The JSON text written by encode_identity.

 required

Returns:

Type Description
dict | None

The identity with its dates and its strike price converted back, or None when the text is not a usable identity.

Example usage

From example_1_keys_and_encodings.py
decoded_identity = self.tier.decode_identity(encoded_identity)
From example_1_keys_and_encodings.py
print(f'Decoding text that is not JSON: {self.tier.decode_identity("not json")}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
def decode_identity(self, encoded):
    """
    Decode one identity dictionary out of the JSON text Redis stores.

    Args:
        encoded (bytes | str): The JSON text written by encode_identity.

    Returns:
        dict | None: The identity with its dates and its strike price converted back, or None when the text is not a usable identity.
    """
    try:
        identity = json.loads(encoded)
    except (ValueError, TypeError):
        return None
    if not isinstance(identity, dict):
        return None

    for field_name in DATE_FIELDS:
        value = identity.get(field_name)
        if value:
            identity[field_name] = datetime.date.fromisoformat(value)
    for field_name in DECIMAL_FIELDS:
        value = identity.get(field_name)
        if value:
            identity[field_name] = decimal.Decimal(value)
    return identity

encode_order_handles 

encode_order_handles(handles_by_broker)

Encode one instrument's order handles as the JSON text Redis stores.

Parameters:

Name Type Description Default
handles_by_broker dict

Mapping of broker name to that broker's handle, itself a dict with keys "broker_token", "order_symbol", "lot_size" and "tick_size".

 required

Returns:

Type Description
str

The handles as JSON.

Example usage

From example_1_keys_and_encodings.py
encoded_handles = self.tier.encode_order_handles(handles)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
740
741
742
743
744
745
746
747
748
749
750
def encode_order_handles(self, handles_by_broker):
    """
    Encode one instrument's order handles as the JSON text Redis stores.

    Args:
        handles_by_broker (dict): Mapping of broker name to that broker's handle, itself a dict with keys "broker_token", "order_symbol", "lot_size" and "tick_size".

    Returns:
        str: The handles as JSON.
    """
    return json.dumps(handles_by_broker)

decode_order_handles 

decode_order_handles(encoded)

Decode one instrument's order handles out of the JSON text Redis stores.

A value that is not a dictionary of dictionaries is rejected rather than returned, so that a Redis left holding an older and narrower shape reads as a miss and is refilled from Postgres.

Parameters:

Name Type Description Default
encoded bytes | str

The JSON text written by encode_order_handles.

 required

Returns:

Type Description
dict | None

Mapping of broker name to that broker's handle, or None when the text is not a usable set of handles.

Example usage

From example_1_keys_and_encodings.py
print(f'Decoded order handles: {self.tier.decode_order_handles(encoded_handles)}')
From example_1_keys_and_encodings.py
print(f'Decoding a bare token in the older shape: {self.tier.decode_order_handles(older_shape)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
def decode_order_handles(self, encoded):
    """
    Decode one instrument's order handles out of the JSON text Redis stores.

    A value that is not a dictionary of dictionaries is rejected rather than returned, so that a Redis left holding an older and narrower shape reads as a miss and is refilled from Postgres.

    Args:
        encoded (bytes | str): The JSON text written by encode_order_handles.

    Returns:
        dict | None: Mapping of broker name to that broker's handle, or None when the text is not a usable set of handles.
    """
    try:
        handles_by_broker = json.loads(encoded)
    except (ValueError, TypeError):
        return None
    if not isinstance(handles_by_broker, dict):
        return None
    for handle in handles_by_broker.values():
        if not isinstance(handle, dict):
            return None
    return handles_by_broker

encode_additional_attributes 

encode_additional_attributes(attributes_by_broker)

Encode one instrument's additional broker attributes as the JSON text Redis stores.

Parameters:

Name Type Description Default
attributes_by_broker dict

Mapping of broker name to that broker's attributes, itself a dict of the names in raw_attributes.ATTRIBUTE_NAMES to a string value or None.

 required

Returns:

Type Description
str

The attributes as JSON.

Example usage

From example_1_keys_and_encodings.py
encoded_attributes = self.tier.encode_additional_attributes(attributes)
From example_2_writing_and_reading_one_day.py
attribute_fields = {
    self.FUTURE_ID: self.tier.encode_additional_attributes({
        'dhan': {
            'freeze_quantity': '1800',
        },
    }),
}
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
775
776
777
778
779
780
781
782
783
784
785
def encode_additional_attributes(self, attributes_by_broker):
    """
    Encode one instrument's additional broker attributes as the JSON text Redis stores.

    Args:
        attributes_by_broker (dict): Mapping of broker name to that broker's attributes, itself a dict of the names in raw_attributes.ATTRIBUTE_NAMES to a string value or None.

    Returns:
        str: The attributes as JSON.
    """
    return json.dumps(attributes_by_broker)

decode_additional_attributes 

decode_additional_attributes(encoded)

Decode one instrument's additional broker attributes out of the JSON text Redis stores.

A value that is not a dictionary of dictionaries reads as a miss, the same way order handles do, so a Redis left holding an older shape is refilled rather than returned.

Parameters:

Name Type Description Default
encoded bytes | str

The JSON text written by encode_additional_attributes.

 required

Returns:

Type Description
dict | None

Mapping of broker name to that broker's attributes, or None when the text is not usable.

Example usage

From example_1_keys_and_encodings.py
print(f'Decoded additional attributes: {self.tier.decode_additional_attributes(encoded_attributes)}')
From example_1_keys_and_encodings.py
print(f'Decoding a list: {self.tier.decode_additional_attributes("[1, 2]")}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
def decode_additional_attributes(self, encoded):
    """
    Decode one instrument's additional broker attributes out of the JSON text Redis stores.

    A value that is not a dictionary of dictionaries reads as a miss, the same way order handles do, so a Redis left holding an older shape is refilled rather than returned.

    Args:
        encoded (bytes | str): The JSON text written by encode_additional_attributes.

    Returns:
        dict | None: Mapping of broker name to that broker's attributes, or None when the text is not usable.
    """
    try:
        attributes_by_broker = json.loads(encoded)
    except (ValueError, TypeError):
        return None
    if not isinstance(attributes_by_broker, dict):
        return None
    for attributes in attributes_by_broker.values():
        if not isinstance(attributes, dict):
            return None
    return attributes_by_broker

read_additional_attributes 

read_additional_attributes(mapping_date, instrument_identifiers)

Read instruments' additional broker attributes out of the additional attribute hash.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required
instrument_identifiers list[str]

The instrument ids to read.

 required

Returns:

Type Description
dict

Mapping of instrument id to a dict of broker name to that broker's attributes. Instruments Redis did not hold, and instruments held in an older shape, are absent.

Example usage

From example_2_writing_and_reading_one_day.py
print(f'read_additional_attributes: {self.tier.read_additional_attributes(self.mapping_date, both)}')
From example_4_when_redis_is_down.py
print(f'read_additional_attributes: {self.tier.read_additional_attributes(self.mapping_date, self.instrument_identifiers)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
def read_additional_attributes(self, mapping_date, instrument_identifiers):
    """
    Read instruments' additional broker attributes out of the additional attribute hash.

    Args:
        mapping_date (datetime.date): The mapping date to read.
        instrument_identifiers (list[str]): The instrument ids to read.

    Returns:
        dict: Mapping of instrument id to a dict of broker name to that broker's attributes. Instruments Redis did not hold, and instruments held in an older shape, are absent.
    """
    client = self.connection.client()
    if client is None:
        return {}
    if not instrument_identifiers:
        return {}

    try:
        stored = client.hmget(self.additional_attributes_key(mapping_date), instrument_identifiers)
    except redis.RedisError:
        return {}

    attributes_by_instrument = {}
    for position, instrument_identifier in enumerate(instrument_identifiers):
        if stored[position] is None:
            continue
        attributes_by_broker = self.decode_additional_attributes(stored[position])
        if attributes_by_broker is not None:
            attributes_by_instrument[instrument_identifier] = attributes_by_broker
    return attributes_by_instrument

has_additional_attributes 

has_additional_attributes(mapping_date)

Say whether the date's additional attribute hash is in Redis at all.

An instrument whose brokers publish nothing extra and a hash that was never warmed both read as no attributes, and only this tells them apart. It is asked only when a read came back empty, which is the rare case, so the extra round trip is not on the ordinary path.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to check.

 required

Returns:

Type Description
bool

True when the hash exists, False when it does not or Redis is unreachable.

Example usage

From example_2_writing_and_reading_one_day.py
print(f'has_additional_attributes today: {self.tier.has_additional_attributes(self.mapping_date)}')
From example_2_writing_and_reading_one_day.py
print(f'has_additional_attributes yesterday: {self.tier.has_additional_attributes(yesterday)}')
From example_4_when_redis_is_down.py
print(f'has_additional_attributes: {self.tier.has_additional_attributes(self.mapping_date)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
def has_additional_attributes(self, mapping_date):
    """
    Say whether the date's additional attribute hash is in Redis at all.

    An instrument whose brokers publish nothing extra and a hash that was never warmed both read as no attributes, and only this tells them apart. It is asked only when a read came back empty, which is the rare case, so the extra round trip is not on the ordinary path.

    Args:
        mapping_date (datetime.date): The mapping date to check.

    Returns:
        bool: True when the hash exists, False when it does not or Redis is unreachable.
    """
    client = self.connection.client()
    if client is None:
        return False
    try:
        return bool(client.exists(self.additional_attributes_key(mapping_date)))
    except redis.RedisError:
        return False

read_current_date 

read_current_date()

The latest warmed mapping date, as Redis holds it.

Returns:

Type Description
date | None

The date, or None when Redis is unreachable or has never been warmed.

Example usage

From example_2_writing_and_reading_one_day.py
print(f'read_current_date: {self.tier.read_current_date()!r}')
From example_4_when_redis_is_down.py
print(f'read_current_date: {self.tier.read_current_date()}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
def read_current_date(self):
    """
    The latest warmed mapping date, as Redis holds it.

    Returns:
        datetime.date | None: The date, or None when Redis is unreachable or has never been warmed.
    """
    client = self.connection.client()
    if client is None:
        return None
    try:
        stored = as_text(client.get(self.current_date_key()))
    except redis.RedisError:
        return None
    if not stored:
        return None
    return datetime.date.fromisoformat(stored)

write_current_date 

write_current_date(mapping_date)

Publish the mapping date the hashes now cover, together with a new warm identifier.

This is written last by a warm, so that a reader sees either the previous complete day or the new complete day and never a half written one. The date and the identifier are set in one transaction, so a reader never sees a new date beside the previous warm's identifier.

Parameters:

Name Type Description Default
mapping_date date

The date to publish.

 required

Returns:

Type Description
bool

True when the write went through, False when Redis was not usable.

Example usage

From example_2_writing_and_reading_one_day.py
print(f'write_current_date: {self.tier.write_current_date(self.mapping_date)}')
From example_4_when_redis_is_down.py
print(f'write_current_date: {self.tier.write_current_date(self.mapping_date)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
def write_current_date(self, mapping_date):
    """
    Publish the mapping date the hashes now cover, together with a new warm identifier.

    This is written last by a warm, so that a reader sees either the previous complete day or the new complete day and never a half written one. The date and the identifier are set in one transaction, so a reader never sees a new date beside the previous warm's identifier.

    Args:
        mapping_date (datetime.date): The date to publish.

    Returns:
        bool: True when the write went through, False when Redis was not usable.
    """
    client = self.connection.client()
    if client is None:
        return False
    try:
        pipeline = client.pipeline(transaction=True)
        pipeline.set(self.current_date_key(), mapping_date.isoformat())
        pipeline.set(self.warm_identifier_key(), uuid.uuid4().hex)
        pipeline.execute()
    except redis.RedisError:
        return False
    return True

read_identities 

read_identities(mapping_date, instrument_identifiers)

Read instruments' identities out of the identity hash.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required
instrument_identifiers list[str]

The instrument ids to read.

 required

Returns:

Type Description
dict

Mapping of instrument id to its identity dict. Instruments Redis did not hold are absent.

Example usage

From example_2_writing_and_reading_one_day.py
identities = self.tier.read_identities(self.mapping_date, both)
From example_4_when_redis_is_down.py
print(f'read_identities: {self.tier.read_identities(self.mapping_date, self.instrument_identifiers)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
def read_identities(self, mapping_date, instrument_identifiers):
    """
    Read instruments' identities out of the identity hash.

    Args:
        mapping_date (datetime.date): The mapping date to read.
        instrument_identifiers (list[str]): The instrument ids to read.

    Returns:
        dict: Mapping of instrument id to its identity dict. Instruments Redis did not hold are absent.
    """
    client = self.connection.client()
    if client is None:
        return {}
    if not instrument_identifiers:
        return {}

    try:
        stored = client.hmget(self.identity_key(mapping_date), instrument_identifiers)
    except redis.RedisError:
        return {}

    identities = {}
    for position, instrument_identifier in enumerate(instrument_identifiers):
        if stored[position] is None:
            continue
        identity = self.decode_identity(stored[position])
        if identity is not None:
            identities[instrument_identifier] = identity
    return identities

write_identities 

write_identities(mapping_date, identities)

Write identities into the identity hash.

Parameters:

Name Type Description Default
mapping_date date

The mapping date the identities belong to.

 required
identities dict

Mapping of instrument id to its identity dict.

 required

Returns:

Type Description
bool

True when the write went through, False when Redis was not usable.

Example usage

From example_2_writing_and_reading_one_day.py
print(f'write_identities: {self.tier.write_identities(self.mapping_date, identities)}')
From example_4_when_redis_is_down.py
print(f'write_identities: {self.tier.write_identities(self.mapping_date, identities)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
def write_identities(self, mapping_date, identities):
    """
    Write identities into the identity hash.

    Args:
        mapping_date (datetime.date): The mapping date the identities belong to.
        identities (dict): Mapping of instrument id to its identity dict.

    Returns:
        bool: True when the write went through, False when Redis was not usable.
    """
    client = self.connection.client()
    if client is None:
        return False
    if not identities:
        return True

    fields = {}
    for instrument_identifier, identity in identities.items():
        fields[instrument_identifier] = self.encode_identity(identity)

    try:
        pipeline = client.pipeline()
        pipeline.hset(self.identity_key(mapping_date), mapping=fields)
        pipeline.expire(self.identity_key(mapping_date), self.seconds_until_midnight())
        pipeline.execute()
    except redis.RedisError:
        return False
    return True

read_token_identifiers 

read_token_identifiers(mapping_date, broker, broker_tokens)

Read one broker's tokens out of the token hash, as the instrument ids each carries.

The identities themselves are not read here, because the caller already knows which of them this process holds and only the rest are worth a round trip.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required
broker str

The broker name, for example "zerodha".

 required
broker_tokens list[str]

The tokens to read.

 required

Returns:

Type Description
dict

Mapping of broker token to a list of instrument ids, in tie-break order. Tokens Redis did not hold are absent.

Example usage

From example_2_writing_and_reading_one_day.py
print(f'read_token_identifiers: {self.tier.read_token_identifiers(self.mapping_date, "zerodha", tokens)}')
From example_4_when_redis_is_down.py
print(f'read_token_identifiers: {self.tier.read_token_identifiers(self.mapping_date, "zerodha", tokens)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
def read_token_identifiers(self, mapping_date, broker, broker_tokens):
    """
    Read one broker's tokens out of the token hash, as the instrument ids each carries.

    The identities themselves are not read here, because the caller already knows which of them this process holds and only the rest are worth a round trip.

    Args:
        mapping_date (datetime.date): The mapping date to read.
        broker (str): The broker name, for example "zerodha".
        broker_tokens (list[str]): The tokens to read.

    Returns:
        dict: Mapping of broker token to a list of instrument ids, in tie-break order. Tokens Redis did not hold are absent.
    """
    client = self.connection.client()
    if client is None:
        return {}
    if not broker_tokens:
        return {}

    try:
        stored = client.hmget(self.tokens_key(mapping_date, broker), broker_tokens)
    except redis.RedisError:
        return {}

    identifiers_by_token = {}
    for position, token in enumerate(broker_tokens):
        joined = as_text(stored[position])
        if joined is None:
            continue
        identifiers_by_token[token] = joined.split(",")
    return identifiers_by_token

write_token_candidates 

write_token_candidates(mapping_date, broker, candidates_by_token)

Write tokens and the identities they carry, in one pipeline.

Both hashes are written together because the token hash holds only instrument ids and is unreadable without the identities they name.

The candidates are stored as their instrument ids joined by commas rather than as JSON, because a UUID contains no comma, the overwhelming majority of tokens have exactly one candidate, and splitting on a comma is cheaper than parsing JSON in the path this exists to make fast.

Parameters:

Name Type Description Default
mapping_date date

The mapping date the answers belong to.

 required
broker str

The broker name, for example "zerodha".

 required
candidates_by_token dict

Mapping of broker token to a list of identity dicts.

 required

Returns:

Type Description
bool

True when the write went through, False when Redis was not usable.

Example usage

From example_2_writing_and_reading_one_day.py
print(f'write_token_candidates: {self.tier.write_token_candidates(self.mapping_date, "zerodha", candidates_by_token)}')
From example_4_when_redis_is_down.py
print(f'write_token_candidates: {self.tier.write_token_candidates(self.mapping_date, "zerodha", {})}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
 997
 998
 999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
def write_token_candidates(self, mapping_date, broker, candidates_by_token):
    """
    Write tokens and the identities they carry, in one pipeline.

    Both hashes are written together because the token hash holds only instrument ids and is unreadable without the identities they name.

    The candidates are stored as their instrument ids joined by commas rather than as JSON, because a UUID contains no comma, the overwhelming majority of tokens have exactly one candidate, and splitting on a comma is cheaper than parsing JSON in the path this exists to make fast.

    Args:
        mapping_date (datetime.date): The mapping date the answers belong to.
        broker (str): The broker name, for example "zerodha".
        candidates_by_token (dict): Mapping of broker token to a list of identity dicts.

    Returns:
        bool: True when the write went through, False when Redis was not usable.
    """
    client = self.connection.client()
    if client is None:
        return False

    token_fields = {}
    identity_fields = {}
    for token, candidates in candidates_by_token.items():
        identifiers = []
        for identity in candidates:
            identifiers.append(identity["instrument_id"])
            identity_fields[identity["instrument_id"]] = self.encode_identity(identity)
        token_fields[token] = ",".join(identifiers)

    if not token_fields:
        return True

    expiry_seconds = self.seconds_until_midnight()
    try:
        pipeline = client.pipeline()
        pipeline.hset(self.tokens_key(mapping_date, broker), mapping=token_fields)
        pipeline.expire(self.tokens_key(mapping_date, broker), expiry_seconds)
        if identity_fields:
            pipeline.hset(self.identity_key(mapping_date), mapping=identity_fields)
            pipeline.expire(self.identity_key(mapping_date), expiry_seconds)
        pipeline.execute()
    except redis.RedisError:
        return False
    return True

read_order_handles 

read_order_handles(mapping_date, instrument_identifiers)

Read instruments' order handles out of the order handle hash.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required
instrument_identifiers list[str]

The instrument ids to read.

 required

Returns:

Type Description
dict

Mapping of instrument id to a dict of broker name to that broker's handle. Instruments Redis did not hold, and instruments held in an older shape, are absent.

Example usage

From example_2_writing_and_reading_one_day.py
handles = self.tier.read_order_handles(self.mapping_date, both)
From example_4_when_redis_is_down.py
print(f'read_order_handles: {self.tier.read_order_handles(self.mapping_date, self.instrument_identifiers)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
def read_order_handles(self, mapping_date, instrument_identifiers):
    """
    Read instruments' order handles out of the order handle hash.

    Args:
        mapping_date (datetime.date): The mapping date to read.
        instrument_identifiers (list[str]): The instrument ids to read.

    Returns:
        dict: Mapping of instrument id to a dict of broker name to that broker's handle. Instruments Redis did not hold, and instruments held in an older shape, are absent.
    """
    client = self.connection.client()
    if client is None:
        return {}
    if not instrument_identifiers:
        return {}

    try:
        stored = client.hmget(self.order_handles_key(mapping_date), instrument_identifiers)
    except redis.RedisError:
        return {}

    handles_by_instrument = {}
    for position, instrument_identifier in enumerate(instrument_identifiers):
        if stored[position] is None:
            continue
        handles_by_broker = self.decode_order_handles(stored[position])
        if handles_by_broker is not None:
            handles_by_instrument[instrument_identifier] = handles_by_broker
    return handles_by_instrument

write_order_handles 

write_order_handles(mapping_date, handles_by_instrument)

Write instruments' order handles into the order handle hash.

This is the only writer of that hash, the daily warm included, which is what keeps the shape written and the shape read the same.

Parameters:

Name Type Description Default
mapping_date date

The mapping date the answers belong to.

 required
handles_by_instrument dict

Mapping of instrument id to a dict of broker name to that broker's handle.

 required

Returns:

Type Description
bool

True when the write went through, False when Redis was not usable.

Example usage

From example_2_writing_and_reading_one_day.py
print(f'write_order_handles: {self.tier.write_order_handles(self.mapping_date, handles_by_instrument)}')
From example_4_when_redis_is_down.py
print(f'write_order_handles: {self.tier.write_order_handles(self.mapping_date, handles)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
def write_order_handles(self, mapping_date, handles_by_instrument):
    """
    Write instruments' order handles into the order handle hash.

    This is the only writer of that hash, the daily warm included, which is what keeps the shape written and the shape read the same.

    Args:
        mapping_date (datetime.date): The mapping date the answers belong to.
        handles_by_instrument (dict): Mapping of instrument id to a dict of broker name to that broker's handle.

    Returns:
        bool: True when the write went through, False when Redis was not usable.
    """
    client = self.connection.client()
    if client is None:
        return False
    if not handles_by_instrument:
        return True

    fields = {}
    for instrument_identifier, handles_by_broker in handles_by_instrument.items():
        fields[instrument_identifier] = self.encode_order_handles(handles_by_broker)

    try:
        pipeline = client.pipeline()
        pipeline.hset(self.order_handles_key(mapping_date), mapping=fields)
        pipeline.expire(self.order_handles_key(mapping_date), self.seconds_until_midnight())
        pipeline.execute()
    except redis.RedisError:
        return False
    return True

write_fields 

write_fields(key, fields, expiry_seconds)

Write one batch of hash fields and set the hash to expire.

This is the batched write a warm streams into, where the expiry is computed once for the whole command rather than per batch.

Parameters:

Name Type Description Default
key str

The hash to write into.

 required
fields dict

The fields to write, as field name to value.

 required
expiry_seconds int

How long the hash may live.

 required

Returns:

Type Description
int

The number of fields written, which is zero when there were none or when Redis was not usable.

Example usage

From example_2_writing_and_reading_one_day.py
written += self.tier.write_fields(self.tier.seen_key(self.mapping_date), seen_fields, expiry_seconds)
From example_2_writing_and_reading_one_day.py
written += self.tier.write_fields(self.tier.underlyings_key(self.mapping_date), underlying_fields, expiry_seconds)
From example_2_writing_and_reading_one_day.py
written += self.tier.write_fields(self.tier.additional_attributes_key(self.mapping_date), attribute_fields, expiry_seconds)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
def write_fields(self, key, fields, expiry_seconds):
    """
    Write one batch of hash fields and set the hash to expire.

    This is the batched write a warm streams into, where the expiry is computed once for the whole command rather than per batch.

    Args:
        key (str): The hash to write into.
        fields (dict): The fields to write, as field name to value.
        expiry_seconds (int): How long the hash may live.

    Returns:
        int: The number of fields written, which is zero when there were none or when Redis was not usable.
    """
    client = self.connection.client()
    if client is None:
        return 0
    if not fields:
        return 0

    try:
        pipeline = client.pipeline()
        pipeline.hset(key, mapping=fields)
        pipeline.expire(key, expiry_seconds)
        pipeline.execute()
    except redis.RedisError:
        return 0
    return len(fields)

clear_other_dates 

clear_other_dates(mapping_date)

Delete every cached mapping date except the one given.

The dated keys expire on their own at the end of the day, so this exists for the case where a warm is re-run for a corrected date and the superseded date's keys would otherwise still be answerable.

Parameters:

Name Type Description Default
mapping_date date

The date to keep.

 required

Returns:

Type Description
int

The number of keys deleted.

Example usage

From example_3_searching_the_catalogue.py
print(f'clear_other_dates deleted: {self.tier.clear_other_dates(self.mapping_date)}')
From example_4_when_redis_is_down.py
print(f'clear_other_dates: {self.tier.clear_other_dates(self.mapping_date)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
def clear_other_dates(self, mapping_date):
    """
    Delete every cached mapping date except the one given.

    The dated keys expire on their own at the end of the day, so this exists for the case where a warm is re-run for a corrected date and the superseded date's keys would otherwise still be answerable.

    Args:
        mapping_date (datetime.date): The date to keep.

    Returns:
        int: The number of keys deleted.
    """
    client = self.connection.client()
    if client is None:
        return 0

    keep = f"{self.KEY_PREFIX}{mapping_date.isoformat()}:"
    deleted = 0
    try:
        for key in client.scan_iter(match=f"{self.KEY_PREFIX}*"):
            name = as_text(key)
            if name == self.current_date_key():
                continue
            if name == self.warm_identifier_key():
                continue
            if name.startswith(keep):
                continue
            client.delete(name)
            deleted += 1
    except redis.RedisError:
        return deleted
    return deleted

MappingPostgresTier 

MappingPostgresTier(engine=None)

Every statement the cache and the warm run against the mapped instrument tables.

The tie-break ordering appears in one place here rather than in three files, because the cached answer and the database answer have to agree field for field and the verification compares them. ORDER BY broker_token, symbol ASC, instrument_id ASC is total only because an instrument id is unique; unified.instruments.symbol is null on every option row, so without the final term Postgres has no defined order among equal keys and one token carrying several options resolves differently from call to call.

Attributes:

Name Type Description
STREAM_ROW_BUFFER int

How many rows a streaming read buffers, so a whole-table pass does not hold the table.
engine Engine

The SQLAlchemy engine over TimescaleDB.

Build the tier around a database engine.

Parameters:

Name Type Description Default
engine Engine

A SQLAlchemy engine over TimescaleDB. None builds one from the environment.

 None

Returns:

Type Description
None

This function returns nothing.

Example programs

examples/stock_brokers/instruments/mapping/utilities/cache/MappingPostgresTier/example_1_point_reads_for_a_few_instruments.py
"""Runs the Postgres tier's point reads, the ones a cache miss falls through to, against rows held in memory.

When neither this process nor Redis holds an answer, the mapping cache asks `MappingPostgresTier`. Each of its point reads runs one statement against `unified.instruments`, `unified.broker_mappings` or `unified.underlyings` and turns the rows into the dictionaries the cache stores: an identity per instrument, a list of candidate identities per broker token, and a handle per broker per instrument.

A small stand-in replaces the SQLAlchemy engine. It holds the rows each statement would return, already in the order the statement's `ORDER BY` asks for, picks them by a fragment of the statement's text, and records the parameters each statement was sent with. That shows the real conversion code at work without a database. Notice that one Dhan security id carries two instruments, because Dhan numbers each exchange's securities separately, and both come back in tie-break order, that instrument ids come back as text although the database hands out UUID objects, and that lot and tick sizes come back as text so they survive being stored as JSON.

Run it from the project root:

    python examples/stock_brokers/instruments/mapping/utilities/cache/MappingPostgresTier/example_1_point_reads_for_a_few_instruments.py
"""

import datetime
import decimal
import types
import uuid

from stock_brokers.instruments.mapping.utilities.cache import (
    MappingPostgresTier,
)


class StandInResult:
    """A stand-in for a SQLAlchemy result holding rows already fetched.

    Attributes:
        rows (list): The rows, in order.
    """

    def __init__(self, rows):
        """Holds the rows.

        Args:
            rows (list): The rows, in order.

        Returns:
            None: This method returns nothing.
        """
        self.rows = rows

    def __iter__(self):
        """Iterates over the rows.

        Returns:
            iterator: An iterator over the rows.
        """
        return iter(self.rows)

    def all(self):
        """Returns every row.

        Returns:
            list: The rows.
        """
        return list(self.rows)

    def scalar(self):
        """Returns the first row's single value.

        Returns:
            object: The first row, which for a scalar answer is the value itself, or None when there is no row.
        """
        if not self.rows:
            return None
        return self.rows[0]


class StandInConnection:
    """A stand-in for a SQLAlchemy connection that asks its engine for each statement's rows.

    Attributes:
        engine (StandInEngine): The engine holding the answers.
    """

    def __init__(self, engine):
        """Builds the connection.

        Args:
            engine (StandInEngine): The engine holding the answers.

        Returns:
            None: This method returns nothing.
        """
        self.engine = engine

    def __enter__(self):
        """Enters a `with` block.

        Returns:
            StandInConnection: This connection.
        """
        return self

    def __exit__(self, exception_type, exception, traceback):
        """Leaves a `with` block without suppressing any exception.

        Args:
            exception_type (type | None): The exception's class, if one was raised.
            exception (BaseException | None): The exception, if one was raised.
            traceback (types.TracebackType | None): The traceback, if one was raised.

        Returns:
            bool: Always False.
        """
        return False

    def execute(self, statement, parameters=None):
        """Answers one statement from the engine's rows.

        Args:
            statement (sqlalchemy.sql.elements.TextClause): The statement.
            parameters (dict | None): The bound parameters.

        Returns:
            StandInResult: The rows for the statement.
        """
        return self.engine.answer(str(statement), parameters)


class StandInEngine:
    """A stand-in for a SQLAlchemy engine that answers statements from rows held in memory.

    Attributes:
        answers (list): Pairs of (text fragment, rows); a statement containing the fragment gets the rows.
        calls (list): Pairs of (text fragment, parameters) for every statement answered.
    """

    def __init__(self):
        """Builds an engine with no answers.

        Returns:
            None: This method returns nothing.
        """
        self.answers = []
        self.calls = []

    def add_answer(self, fragment, rows):
        """Adds the rows a statement containing a fragment should return.

        Args:
            fragment (str): Text that appears in the statement and nowhere in the others.
            rows (list): The rows to return.

        Returns:
            None: This method returns nothing.
        """
        self.answers.append((fragment, rows))

    def connect(self):
        """Opens a connection.

        Returns:
            StandInConnection: A new connection.
        """
        return StandInConnection(self)

    def answer(self, statement_text, parameters):
        """Finds the rows for one statement and records the call.

        Args:
            statement_text (str): The statement's SQL.
            parameters (dict | None): The bound parameters.

        Returns:
            StandInResult: The rows for the statement.

        Raises:
            ValueError: If no fragment matches the statement.
        """
        for fragment, rows in self.answers:
            if fragment in statement_text:
                self.calls.append((fragment, parameters))
                return StandInResult(rows)
        raise ValueError(f'No stand-in answer for statement: {statement_text}')


class PointReadsExample:
    """Runs each point read of the Postgres tier and prints what it built from the rows.

    Attributes:
        engine (StandInEngine): The stand-in engine holding the rows.
        tier (MappingPostgresTier): The tier being shown.
        mapping_date (datetime.date): The mapping date asked about.
    """

    INFY_ID = uuid.UUID('0a6e2d41-9c7f-4b58-8d13-6f2e4a9b7c05')
    INFY_BSE_ID = uuid.UUID('3d5f7a9c-1e2b-4d6f-8a0c-2e4f6a8c0e11')
    NIFTY_FUTURE_ID = uuid.UUID('7c1e9b20-4d3a-4f87-a6b5-0e2d9c8f1a44')
    NIFTY_INDEX_ID = uuid.UUID('2b9d4e6f-8a1c-4e35-b7d2-5f0a3c6e9b18')

    def __init__(self):
        """Builds the engine's rows and the tier around it.

        Returns:
            None: This method returns nothing.
        """
        self.mapping_date = datetime.date(2026, 9, 30)
        self.engine = StandInEngine()
        self.engine.add_answer('max(mapping_date)', [
            self.mapping_date,
        ])
        self.engine.add_answer('b.broker_token = ANY', [
            self.token_row('1594', self.INFY_ID, 'nse', 'nse_equities'),
            self.token_row('1594', self.INFY_BSE_ID, 'bse', 'bse_equities'),
        ])
        self.engine.add_answer('SELECT instrument_id, exchange', [
            self.instrument_row(self.NIFTY_FUTURE_ID, 'nse', 'nse_equity_index_futures', 'future', None, 'NIFTY', datetime.date(2026, 10, 27)),
        ])
        self.engine.add_answer('SELECT instrument_id, broker, broker_token', [
            self.handle_row(self.NIFTY_FUTURE_ID, 'dhan', '52168', None, decimal.Decimal('75'), decimal.Decimal('0.10')),
            self.handle_row(self.NIFTY_FUTURE_ID, 'zerodha', '13238018', 'NIFTY26OCTFUT', decimal.Decimal('75'), decimal.Decimal('0.10')),
        ])
        self.engine.add_answer('SELECT DISTINCT broker_token', [
            types.SimpleNamespace(broker_token='1594'),
            types.SimpleNamespace(broker_token='13238018'),
        ])
        self.engine.add_answer('FROM unified.underlyings', [
            types.SimpleNamespace(instrument_id=self.NIFTY_FUTURE_ID, underlying_instrument_id=self.NIFTY_INDEX_ID),
        ])
        self.engine.add_answer('SELECT instrument_id, broker, attributes', [
            types.SimpleNamespace(instrument_id=self.INFY_ID, broker='dhan', attributes=self.equity_attributes()),
            types.SimpleNamespace(instrument_id=self.INFY_ID, broker='fyers', attributes=self.equity_attributes()),
        ])
        self.tier = MappingPostgresTier(self.engine)

    def equity_attributes(self):
        """Builds the additional attributes a broker publishes for INFY, cut down to two.

        Returns:
            dict: The attributes.
        """
        return {
            'freeze_quantity': None,
            'isin': 'INE009A01021',
        }

    def instrument_row(self, instrument_identifier, exchange, segment, shape, symbol, underlying_symbol, expiry_date):
        """Builds one row of `unified.instruments` with no strike or option type.

        Args:
            instrument_identifier (uuid.UUID): The instrument id.
            exchange (str): The exchange.
            segment (str): The exchange-prefixed segment.
            shape (str): "security", "future" or "option".
            symbol (str | None): The symbol of a security.
            underlying_symbol (str | None): The underlying of a derivative.
            expiry_date (datetime.date | None): The expiry of a derivative.

        Returns:
            types.SimpleNamespace: The row.
        """
        return types.SimpleNamespace(
            instrument_id=instrument_identifier,
            exchange=exchange,
            segment=segment,
            shape=shape,
            symbol=symbol,
            underlying_symbol=underlying_symbol,
            expiry_date=expiry_date,
            strike_price=None,
            option_type=None,
        )

    def token_row(self, broker_token, instrument_identifier, exchange, segment):
        """Builds one row of the token candidate join for an equity.

        Args:
            broker_token (str): The broker's token.
            instrument_identifier (uuid.UUID): The instrument id.
            exchange (str): The exchange.
            segment (str): The exchange-prefixed segment.

        Returns:
            types.SimpleNamespace: The row.
        """
        row = self.instrument_row(instrument_identifier, exchange, segment, 'security', 'INFY', None, None)
        row.broker_token = broker_token
        row.mapping_date = self.mapping_date
        return row

    def handle_row(self, instrument_identifier, broker, broker_token, order_symbol, lot_size, tick_size):
        """Builds one row of `unified.broker_mappings` carrying an order handle.

        Args:
            instrument_identifier (uuid.UUID): The instrument id.
            broker (str): The broker name.
            broker_token (str): The broker's token.
            order_symbol (str | None): The symbol the broker orders by, if any.
            lot_size (decimal.Decimal): The lot size.
            tick_size (decimal.Decimal): The tick size.

        Returns:
            types.SimpleNamespace: The row.
        """
        return types.SimpleNamespace(
            instrument_id=instrument_identifier,
            broker=broker,
            broker_token=broker_token,
            order_symbol=order_symbol,
            lot_size=lot_size,
            tick_size=tick_size,
        )

    def run(self):
        """Runs each point read and prints its answer, then the parameters sent.

        Returns:
            None: This method returns nothing.
        """
        print(f'latest_mapping_date: {self.tier.latest_mapping_date()!r}')
        tokens = [
            '1594',
        ]
        candidates = self.tier.read_token_candidates(self.mapping_date, 'dhan', tokens)
        for identity in candidates['1594']:
            print(f'read_token_candidates 1594: {identity["segment"]} {identity["instrument_id"]}')
        future_text = str(self.NIFTY_FUTURE_ID)
        future_only = [
            future_text,
        ]
        identities = self.tier.read_identities(self.mapping_date, future_only)
        print(f'read_identities: {identities[future_text]}')
        handles = self.tier.read_order_handles(self.mapping_date, future_only)
        for broker, handle in handles[future_text].items():
            print(f'read_order_handles {broker}: {handle}')
        print(f'broker_tokens_on_date: {self.tier.broker_tokens_on_date(self.mapping_date, "zerodha")}')
        print(f'read_underlyings: {self.tier.read_underlyings(self.mapping_date, future_only)}')
        print(f'read_underlyings of nothing: {self.tier.read_underlyings(self.mapping_date, [])}')
        infy_only = [
            str(self.INFY_ID),
        ]
        print(f'read_additional_attributes: {self.tier.read_additional_attributes(self.mapping_date, infy_only)}')
        row = self.handle_row(self.NIFTY_FUTURE_ID, 'kotak', '35012', 'NIFTY26OCTFUT', decimal.Decimal('75'), None)
        print(f'handle_from_row with no tick size: {self.tier.handle_from_row(row)}')
        row = self.instrument_row(self.NIFTY_INDEX_ID, 'nse', 'nse_equity_indices', 'security', 'NIFTY 50', None, None)
        print(f'identity_from_row: {self.tier.identity_from_row(row, self.mapping_date)}')
        print('Statements sent, with their parameters:')
        for fragment, parameters in self.engine.calls:
            print(f'  {fragment}: {parameters}')


if __name__ == '__main__':
    PointReadsExample().run()

It prints:

latest_mapping_date: datetime.date(2026, 9, 30)
read_token_candidates 1594: nse_equities 0a6e2d41-9c7f-4b58-8d13-6f2e4a9b7c05
read_token_candidates 1594: bse_equities 3d5f7a9c-1e2b-4d6f-8a0c-2e4f6a8c0e11
read_identities: {'instrument_id': '7c1e9b20-4d3a-4f87-a6b5-0e2d9c8f1a44', 'exchange': 'nse', 'segment': 'nse_equity_index_futures', 'shape': 'future', 'symbol': None, 'underlying_symbol': 'NIFTY', 'expiry_date': datetime.date(2026, 10, 27), 'strike_price': None, 'option_type': None, 'mapping_date': datetime.date(2026, 9, 30)}
read_order_handles dhan: {'broker_token': '52168', 'order_symbol': None, 'lot_size': '75', 'tick_size': '0.10'}
read_order_handles zerodha: {'broker_token': '13238018', 'order_symbol': 'NIFTY26OCTFUT', 'lot_size': '75', 'tick_size': '0.10'}
broker_tokens_on_date: ['1594', '13238018']
read_underlyings: {'7c1e9b20-4d3a-4f87-a6b5-0e2d9c8f1a44': '2b9d4e6f-8a1c-4e35-b7d2-5f0a3c6e9b18'}
read_underlyings of nothing: {}
read_additional_attributes: {'0a6e2d41-9c7f-4b58-8d13-6f2e4a9b7c05': {'dhan': {'freeze_quantity': None, 'isin': 'INE009A01021'}, 'fyers': {'freeze_quantity': None, 'isin': 'INE009A01021'}}}
handle_from_row with no tick size: {'broker_token': '35012', 'order_symbol': 'NIFTY26OCTFUT', 'lot_size': '75', 'tick_size': None}
identity_from_row: {'instrument_id': '2b9d4e6f-8a1c-4e35-b7d2-5f0a3c6e9b18', 'exchange': 'nse', 'segment': 'nse_equity_indices', 'shape': 'security', 'symbol': 'NIFTY 50', 'underlying_symbol': None, 'expiry_date': None, 'strike_price': None, 'option_type': None, 'mapping_date': datetime.date(2026, 9, 30)}
Statements sent, with their parameters:
  max(mapping_date): None
  b.broker_token = ANY: {'broker': 'dhan', 'mapping_date': datetime.date(2026, 9, 30), 'tokens': ['1594']}
  SELECT instrument_id, exchange: {'instrument_identifiers': ['7c1e9b20-4d3a-4f87-a6b5-0e2d9c8f1a44']}
  SELECT instrument_id, broker, broker_token: {'instrument_identifiers': ['7c1e9b20-4d3a-4f87-a6b5-0e2d9c8f1a44'], 'mapping_date': datetime.date(2026, 9, 30)}
  SELECT DISTINCT broker_token: {'broker': 'zerodha', 'mapping_date': datetime.date(2026, 9, 30)}
  FROM unified.underlyings: {'mapping_date': datetime.date(2026, 9, 30), 'instrument_ids': ['7c1e9b20-4d3a-4f87-a6b5-0e2d9c8f1a44']}
  SELECT instrument_id, broker, attributes: {'mapping_date': datetime.date(2026, 9, 30), 'instrument_identifiers': ['0a6e2d41-9c7f-4b58-8d13-6f2e4a9b7c05']}
examples/stock_brokers/instruments/mapping/utilities/cache/MappingPostgresTier/example_2_streaming_a_whole_day.py
"""Streams a whole day's mapping out of Postgres the way the daily Redis warm does, one row at a time.

The warm that fills Redis each morning reads every mapped instrument, every broker token, every order handle, every contract size, every resolved underlying and every broker's additional attributes. Holding all of that at once would cost hundreds of megabytes, so each `stream_*` method opens its connection through `streaming_connection()`, which asks the driver to stream results with a bounded row buffer, and yields one converted row at a time.

A small stand-in replaces the SQLAlchemy engine. It holds the rows each statement would return, already in the statement's `ORDER BY` order, picks them by a fragment of the statement's text, and records the execution options each connection was given. The last part of the program uses a second stand-in engine whose `unified.underlyings` table does not exist yet, as on a database where that DDL has not been applied, and shows `read_underlyings` answering an empty dictionary rather than raising.

Notice that the streamed identities carry the mapping date the caller passed, that one token's rows arrive together, and that every streaming connection was opened with `stream_results` and a buffer of `STREAM_ROW_BUFFER` rows.

Run it from the project root:

    python examples/stock_brokers/instruments/mapping/utilities/cache/MappingPostgresTier/example_2_streaming_a_whole_day.py
"""

import datetime
import decimal
import types
import uuid

from sqlalchemy.exc import ProgrammingError

from stock_brokers.instruments.mapping.utilities.cache import (
    MappingPostgresTier,
)


class StandInResult:
    """A stand-in for a SQLAlchemy result holding rows already fetched.

    Attributes:
        rows (list): The rows, in order.
    """

    def __init__(self, rows):
        """Holds the rows.

        Args:
            rows (list): The rows, in order.

        Returns:
            None: This method returns nothing.
        """
        self.rows = rows

    def __iter__(self):
        """Iterates over the rows.

        Returns:
            iterator: An iterator over the rows.
        """
        return iter(self.rows)

    def all(self):
        """Returns every row.

        Returns:
            list: The rows.
        """
        return list(self.rows)

    def scalar(self):
        """Returns the first row's single value.

        Returns:
            object: The first row, which for a scalar answer is the value itself, or None when there is no row.
        """
        if not self.rows:
            return None
        return self.rows[0]


class StandInConnection:
    """A stand-in for a SQLAlchemy connection that asks its engine for each statement's rows.

    Attributes:
        engine (StandInEngine): The engine holding the answers.
    """

    def __init__(self, engine):
        """Builds the connection.

        Args:
            engine (StandInEngine): The engine holding the answers.

        Returns:
            None: This method returns nothing.
        """
        self.engine = engine

    def __enter__(self):
        """Enters a `with` block.

        Returns:
            StandInConnection: This connection.
        """
        return self

    def __exit__(self, exception_type, exception, traceback):
        """Leaves a `with` block without suppressing any exception.

        Args:
            exception_type (type | None): The exception's class, if one was raised.
            exception (BaseException | None): The exception, if one was raised.
            traceback (types.TracebackType | None): The traceback, if one was raised.

        Returns:
            bool: Always False.
        """
        return False

    def execution_options(self, **options):
        """Records the execution options a caller sets and returns this connection.

        Args:
            **options (dict): The options, such as stream_results.

        Returns:
            StandInConnection: This connection.
        """
        self.engine.options.append(options)
        return self

    def execute(self, statement, parameters=None):
        """Answers one statement from the engine's rows.

        Args:
            statement (sqlalchemy.sql.elements.TextClause): The statement.
            parameters (dict | None): The bound parameters.

        Returns:
            StandInResult: The rows for the statement.
        """
        return self.engine.answer(str(statement), parameters)


class StandInEngine:
    """A stand-in for a SQLAlchemy engine that answers statements from rows held in memory.

    Attributes:
        answers (list): Pairs of (text fragment, rows); a statement containing the fragment gets the rows, or has the rows raised when they are an exception.
        calls (list): Pairs of (text fragment, parameters) for every statement answered.
        options (list): The execution options set on each connection.
    """

    def __init__(self):
        """Builds an engine with no answers.

        Returns:
            None: This method returns nothing.
        """
        self.answers = []
        self.calls = []
        self.options = []

    def add_answer(self, fragment, rows):
        """Adds the rows a statement containing a fragment should return.

        Args:
            fragment (str): Text that appears in the statement and nowhere in the others.
            rows (list | Exception): The rows to return, or an exception to raise.

        Returns:
            None: This method returns nothing.
        """
        self.answers.append((fragment, rows))

    def connect(self):
        """Opens a connection.

        Returns:
            StandInConnection: A new connection.
        """
        return StandInConnection(self)

    def answer(self, statement_text, parameters):
        """Finds the rows for one statement and records the call.

        Args:
            statement_text (str): The statement's SQL.
            parameters (dict | None): The bound parameters.

        Returns:
            StandInResult: The rows for the statement.

        Raises:
            ValueError: If no fragment matches the statement.
            Exception: The exception stored as the answer, when one is.
        """
        for fragment, rows in self.answers:
            if fragment in statement_text:
                self.calls.append((fragment, parameters))
                if isinstance(rows, Exception):
                    raise rows
                return StandInResult(rows)
        raise ValueError(f'No stand-in answer for statement: {statement_text}')


class StreamingAWholeDayExample:
    """Streams each kind of row the warm reads and prints what the tier yields.

    Attributes:
        engine (StandInEngine): The stand-in engine holding the day's rows.
        tier (MappingPostgresTier): The tier being shown.
        mapping_date (datetime.date): The mapping date streamed.
    """

    INFY_ID = uuid.UUID('0a6e2d41-9c7f-4b58-8d13-6f2e4a9b7c05')
    NIFTY_FUTURE_ID = uuid.UUID('7c1e9b20-4d3a-4f87-a6b5-0e2d9c8f1a44')
    NIFTY_INDEX_ID = uuid.UUID('2b9d4e6f-8a1c-4e35-b7d2-5f0a3c6e9b18')

    def __init__(self):
        """Builds the engine's rows and the tier around it.

        Returns:
            None: This method returns nothing.
        """
        self.mapping_date = datetime.date(2026, 9, 30)
        self.engine = StandInEngine()
        self.engine.add_answer('m.last_seen_date', [
            self.catalogue_row(self.INFY_ID, 'nse_equities', 'INFY', None, datetime.date(2024, 1, 2)),
            self.catalogue_row(self.NIFTY_FUTURE_ID, 'nse_equity_index_futures', None, 'NIFTY', datetime.date(2026, 7, 1)),
        ])
        self.engine.add_answer('m.option_type FROM', [
            self.catalogue_row(self.INFY_ID, 'nse_equities', 'INFY', None, datetime.date(2024, 1, 2)),
            self.catalogue_row(self.NIFTY_FUTURE_ID, 'nse_equity_index_futures', None, 'NIFTY', datetime.date(2026, 7, 1)),
        ])
        self.engine.add_answer('SELECT b.broker_token, m.instrument_id', [
            types.SimpleNamespace(broker_token='13238018', instrument_id=self.NIFTY_FUTURE_ID),
            types.SimpleNamespace(broker_token='408065', instrument_id=self.INFY_ID),
        ])
        self.engine.add_answer('SELECT instrument_id, broker, broker_token', [
            self.handle_row(self.INFY_ID, 'dhan', '1594', None, '1', '0.10'),
            self.handle_row(self.INFY_ID, 'zerodha', '408065', 'INFY', '1', '0.10'),
            self.handle_row(self.NIFTY_FUTURE_ID, 'zerodha', '13238018', 'NIFTY26OCTFUT', '75', '0.10'),
        ])
        self.engine.add_answer('FROM unified.contract_sizes', [
            types.SimpleNamespace(instrument_id=self.NIFTY_FUTURE_ID, units_per_lot=decimal.Decimal('75'), status='confirmed', tradeable=True),
        ])
        self.engine.add_answer('FROM unified.underlyings', [
            types.SimpleNamespace(instrument_id=self.NIFTY_FUTURE_ID, underlying_instrument_id=self.NIFTY_INDEX_ID),
        ])
        self.engine.add_answer('SELECT instrument_id, broker, attributes', [
            types.SimpleNamespace(instrument_id=self.INFY_ID, broker='dhan', attributes=self.dhan_attributes()),
        ])
        self.tier = MappingPostgresTier(self.engine)

    def dhan_attributes(self):
        """Builds Dhan's additional attributes for INFY, cut down to two.

        Returns:
            dict: The attributes.
        """
        return {
            'isin': 'INE009A01021',
            'series': 'EQ',
        }

    def catalogue_row(self, instrument_identifier, segment, symbol, underlying_symbol, first_seen_date):
        """Builds one row of the catalogue join, which also serves the identity stream.

        Args:
            instrument_identifier (uuid.UUID): The instrument id.
            segment (str): The exchange-prefixed segment.
            symbol (str | None): The symbol of a security.
            underlying_symbol (str | None): The underlying of a future.
            first_seen_date (datetime.date): The first date the instrument was mapped.

        Returns:
            types.SimpleNamespace: The row.
        """
        shape = 'security'
        expiry_date = None
        if underlying_symbol is not None:
            shape = 'future'
            expiry_date = datetime.date(2026, 10, 27)
        return types.SimpleNamespace(
            instrument_id=instrument_identifier,
            exchange='nse',
            segment=segment,
            shape=shape,
            symbol=symbol,
            underlying_symbol=underlying_symbol,
            expiry_date=expiry_date,
            strike_price=None,
            option_type=None,
            first_seen_date=first_seen_date,
            last_seen_date=self.mapping_date,
        )

    def handle_row(self, instrument_identifier, broker, broker_token, order_symbol, lot_size, tick_size):
        """Builds one row of `unified.broker_mappings` carrying an order handle.

        Args:
            instrument_identifier (uuid.UUID): The instrument id.
            broker (str): The broker name.
            broker_token (str): The broker's token.
            order_symbol (str | None): The symbol the broker orders by, if any.
            lot_size (str): The lot size, as text.
            tick_size (str): The tick size, as text.

        Returns:
            types.SimpleNamespace: The row.
        """
        return types.SimpleNamespace(
            instrument_id=instrument_identifier,
            broker=broker,
            broker_token=broker_token,
            order_symbol=order_symbol,
            lot_size=decimal.Decimal(lot_size),
            tick_size=decimal.Decimal(tick_size),
        )

    def stream_everything(self):
        """Runs each stream and prints what it yields.

        Returns:
            None: This method returns nothing.
        """
        for identity in self.tier.stream_identities(self.mapping_date):
            print(f'stream_identities: {identity["segment"]} {identity["symbol"] or identity["underlying_symbol"]} mapped on {identity["mapping_date"]}')
        for broker_token, instrument_identifier in self.tier.stream_broker_tokens(self.mapping_date, 'zerodha'):
            print(f'stream_broker_tokens zerodha: {broker_token} -> {instrument_identifier}')
        for instrument_identifier, broker, handle in self.tier.stream_order_handles(self.mapping_date):
            print(f'stream_order_handles: {instrument_identifier} {broker} {handle}')
        for instrument_identifier, units_per_lot, status, tradeable in self.tier.stream_contract_sizes(self.mapping_date):
            print(f'stream_contract_sizes: {instrument_identifier} {units_per_lot} {status} tradeable={tradeable}')
        for instrument_identifier, underlying_identifier in self.tier.stream_underlyings(self.mapping_date):
            print(f'stream_underlyings: {instrument_identifier} is written on {underlying_identifier}')
        for instrument_identifier, broker, attributes in self.tier.stream_additional_attributes(self.mapping_date):
            print(f'stream_additional_attributes: {instrument_identifier} {broker} {attributes}')
        for identity, first_seen_date, last_seen_date in self.tier.stream_catalogue(self.mapping_date):
            print(f'stream_catalogue: {identity["instrument_id"]} seen {first_seen_date} to {last_seen_date}')

    def show_streaming_connection(self):
        """Opens one streaming connection directly and prints the options every stream used.

        Returns:
            None: This method returns nothing.
        """
        with self.tier.streaming_connection() as connection:
            print(f'streaming_connection gives a {type(connection).__name__}')
        print(f'Connections opened with streaming options: {len(self.engine.options)}')
        print(f'The options: {self.engine.options[0]}')

    def show_missing_underlyings_table(self):
        """Reads underlyings from a database where that table does not exist yet.

        Returns:
            None: This method returns nothing.
        """
        missing_table = ProgrammingError(
            'SELECT instrument_id, underlying_instrument_id FROM unified.underlyings',
            {},
            Exception('relation "unified.underlyings" does not exist'),
        )
        fresh_engine = StandInEngine()
        fresh_engine.add_answer('FROM unified.underlyings', missing_table)
        fresh_tier = MappingPostgresTier(fresh_engine)
        wanted = [
            str(self.NIFTY_FUTURE_ID),
        ]
        print(f'read_underlyings before the table exists: {fresh_tier.read_underlyings(self.mapping_date, wanted)}')

    def run(self):
        """Streams the day, shows the connection options, then the missing table.

        Returns:
            None: This method returns nothing.
        """
        print(f'Rows buffered per streaming read: {MappingPostgresTier.STREAM_ROW_BUFFER}')
        self.stream_everything()
        self.show_streaming_connection()
        self.show_missing_underlyings_table()


if __name__ == '__main__':
    StreamingAWholeDayExample().run()

It prints:

Rows buffered per streaming read: 20000
stream_identities: nse_equities INFY mapped on 2026-09-30
stream_identities: nse_equity_index_futures NIFTY mapped on 2026-09-30
stream_broker_tokens zerodha: 13238018 -> 7c1e9b20-4d3a-4f87-a6b5-0e2d9c8f1a44
stream_broker_tokens zerodha: 408065 -> 0a6e2d41-9c7f-4b58-8d13-6f2e4a9b7c05
stream_order_handles: 0a6e2d41-9c7f-4b58-8d13-6f2e4a9b7c05 dhan {'broker_token': '1594', 'order_symbol': None, 'lot_size': '1', 'tick_size': '0.10'}
stream_order_handles: 0a6e2d41-9c7f-4b58-8d13-6f2e4a9b7c05 zerodha {'broker_token': '408065', 'order_symbol': 'INFY', 'lot_size': '1', 'tick_size': '0.10'}
stream_order_handles: 7c1e9b20-4d3a-4f87-a6b5-0e2d9c8f1a44 zerodha {'broker_token': '13238018', 'order_symbol': 'NIFTY26OCTFUT', 'lot_size': '75', 'tick_size': '0.10'}
stream_contract_sizes: 7c1e9b20-4d3a-4f87-a6b5-0e2d9c8f1a44 75 confirmed tradeable=True
stream_underlyings: 7c1e9b20-4d3a-4f87-a6b5-0e2d9c8f1a44 is written on 2b9d4e6f-8a1c-4e35-b7d2-5f0a3c6e9b18
stream_additional_attributes: 0a6e2d41-9c7f-4b58-8d13-6f2e4a9b7c05 dhan {'isin': 'INE009A01021', 'series': 'EQ'}
stream_catalogue: 0a6e2d41-9c7f-4b58-8d13-6f2e4a9b7c05 seen 2024-01-02 to 2026-09-30
stream_catalogue: 7c1e9b20-4d3a-4f87-a6b5-0e2d9c8f1a44 seen 2026-07-01 to 2026-09-30
streaming_connection gives a StandInConnection
Connections opened with streaming options: 8
The options: {'stream_results': True, 'max_row_buffer': 20000}
read_underlyings before the table exists: {}

Methods:

Name Description
latest_mapping_date

Read the latest mapping date straight from Postgres.
read_token_candidates

Read one broker's tokens, with every candidate identity each token carries.
read_identities

Read instruments' identities straight out of unified.instruments.
read_order_handles

Read instruments' order handles for one mapping date.
broker_tokens_on_date

Every token one broker maps on one date, which is what a whole-broker warm has to resolve.
stream_identities

Yield the identity of every instrument mapped on the date, one at a time.
stream_broker_tokens

Yield one broker's token and instrument id pairs for the date, ordered so one token's rows arrive together.
stream_order_handles

Yield every instrument's order handles for the date, ordered so one instrument's rows arrive together.
stream_contract_sizes

Yield every contract size decision for the date.
stream_underlyings

Yield every resolved underlying for the date.
read_underlyings

Read instruments' resolved underlyings straight out of unified.underlyings.
stream_additional_attributes

Yield every instrument's additional broker attributes for the date, ordered so one instrument's rows arrive together.
read_additional_attributes

Read instruments' additional broker attributes straight from the mapping table.
stream_catalogue

Yield the identity and seen dates of every instrument mapped on the date, one at a time.
streaming_connection

A connection that streams its results rather than buffering the whole answer.
identity_from_row

Build one identity dictionary from a database row.
handle_from_row

Build one broker's order handle from a database row.
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
def __init__(self, engine=None):
    """
    Build the tier around a database engine.

    Args:
        engine (sqlalchemy.engine.Engine): A SQLAlchemy engine over TimescaleDB. None builds one from the environment.

    Returns:
        None: This function returns nothing.
    """
    self.engine = engine or get_postgres_engine()

latest_mapping_date 

latest_mapping_date()

Read the latest mapping date straight from Postgres.

This costs about 1.2 seconds, because a max over the whole hypertable has no selective filter of its own and scans every chunk. It is the fallback for a Redis that has never been warmed, never the read path.

Returns:

Type Description
date | None

The latest mapping date, or None when nothing has been mapped.

Example usage

From example_1_point_reads_for_a_few_instruments.py
print(f'latest_mapping_date: {self.tier.latest_mapping_date()!r}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
def latest_mapping_date(self):
    """
    Read the latest mapping date straight from Postgres.

    This costs about 1.2 seconds, because a max over the whole hypertable has no selective filter of its own and scans every chunk. It is the fallback for a Redis that has never been warmed, never the read path.

    Returns:
        datetime.date | None: The latest mapping date, or None when nothing has been mapped.
    """
    with self.engine.connect() as connection:
        return connection.execute(
            text(f"SELECT max(mapping_date) FROM {tables.BROKER_MAPPINGS}")
        ).scalar()

read_token_candidates 

read_token_candidates(mapping_date, broker, broker_tokens)

Read one broker's tokens, with every candidate identity each token carries.

The mapping date is bound rather than derived, because a max over the hypertable is a hundred times more expensive than the look-up it would serve.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required
broker str

The broker name, for example "zerodha".

 required
broker_tokens list[str]

The tokens to read.

 required

Returns:

Type Description
dict

Mapping of broker token to a list of identity dicts, in tie-break order. Tokens with no mapping on the date are absent.

Example usage

From example_1_point_reads_for_a_few_instruments.py
candidates = self.tier.read_token_candidates(self.mapping_date, 'dhan', tokens)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
def read_token_candidates(self, mapping_date, broker, broker_tokens):
    """
    Read one broker's tokens, with every candidate identity each token carries.

    The mapping date is bound rather than derived, because a max over the hypertable is a hundred times more expensive than the look-up it would serve.

    Args:
        mapping_date (datetime.date): The mapping date to read.
        broker (str): The broker name, for example "zerodha".
        broker_tokens (list[str]): The tokens to read.

    Returns:
        dict: Mapping of broker token to a list of identity dicts, in tie-break order. Tokens with no mapping on the date are absent.
    """
    with self.engine.connect() as connection:
        rows = connection.execute(
            text(
                "SELECT b.broker_token, b.mapping_date, m.instrument_id, m.exchange, m.segment, m.shape, "
                "  m.symbol, m.underlying_symbol, m.expiry_date, m.strike_price, m.option_type "
                f"FROM {tables.BROKER_MAPPINGS} b "
                f"JOIN {tables.MASTER} m ON m.instrument_id = b.instrument_id "
                "WHERE b.broker = :broker AND b.mapping_date = :mapping_date "
                "  AND b.broker_token = ANY(:tokens) "
                "ORDER BY b.broker_token, m.symbol ASC, m.instrument_id ASC"
            ),
            {
                "broker": broker,
                "mapping_date": mapping_date,
                "tokens": broker_tokens,
            },
        ).all()

    candidates_by_token = {}
    for row in rows:
        candidates_by_token.setdefault(row.broker_token, []).append(self.identity_from_row(row, row.mapping_date))
    return candidates_by_token

read_identities 

read_identities(mapping_date, instrument_identifiers)

Read instruments' identities straight out of unified.instruments.

The mapping date is taken rather than derived, and is used only to stamp the identity, because unified.instruments is not a hypertable and carries one row per instrument for all time.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to stamp on each identity.

 required
instrument_identifiers list[str]

The instrument ids to read.

 required

Returns:

Type Description
dict

Mapping of instrument id to its identity dict. Instruments the master does not hold are absent.

Example usage

From example_1_point_reads_for_a_few_instruments.py
identities = self.tier.read_identities(self.mapping_date, future_only)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
def read_identities(self, mapping_date, instrument_identifiers):
    """
    Read instruments' identities straight out of unified.instruments.

    The mapping date is taken rather than derived, and is used only to stamp the identity, because unified.instruments is not a hypertable and carries one row per instrument for all time.

    Args:
        mapping_date (datetime.date): The mapping date to stamp on each identity.
        instrument_identifiers (list[str]): The instrument ids to read.

    Returns:
        dict: Mapping of instrument id to its identity dict. Instruments the master does not hold are absent.
    """
    with self.engine.connect() as connection:
        rows = connection.execute(
            text(
                "SELECT instrument_id, exchange, segment, shape, symbol, underlying_symbol, "
                "  expiry_date, strike_price, option_type "
                f"FROM {tables.MASTER} "
                "WHERE instrument_id = ANY(CAST(:instrument_identifiers AS uuid[]))"
            ),
            {
                "instrument_identifiers": instrument_identifiers,
            },
        ).all()

    identities = {}
    for row in rows:
        identities[str(row.instrument_id)] = self.identity_from_row(row, mapping_date)
    return identities

read_order_handles 

read_order_handles(mapping_date, instrument_identifiers)

Read instruments' order handles for one mapping date.

Four columns are read rather than one because everything an order needs about an instrument at a broker is here, and reading the token now and the symbol, lot size and tick size later would be two queries for one answer on a path that must not make even one.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required
instrument_identifiers list[str]

The instrument ids to read.

 required

Returns:

Type Description
dict

Mapping of instrument id to a dict of broker name to that broker's handle. Instruments with no mapping on the date are absent.

Example usage

From example_1_point_reads_for_a_few_instruments.py
handles = self.tier.read_order_handles(self.mapping_date, future_only)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
1303
1304
1305
1306
def read_order_handles(self, mapping_date, instrument_identifiers):
    """
    Read instruments' order handles for one mapping date.

    Four columns are read rather than one because everything an order needs about an instrument at a broker is here, and reading the token now and the symbol, lot size and tick size later would be two queries for one answer on a path that must not make even one.

    Args:
        mapping_date (datetime.date): The mapping date to read.
        instrument_identifiers (list[str]): The instrument ids to read.

    Returns:
        dict: Mapping of instrument id to a dict of broker name to that broker's handle. Instruments with no mapping on the date are absent.
    """
    with self.engine.connect() as connection:
        rows = connection.execute(
            text(
                "SELECT instrument_id, broker, broker_token, order_symbol, lot_size, tick_size "
                f"FROM {tables.BROKER_MAPPINGS} "
                "WHERE instrument_id = ANY(CAST(:instrument_identifiers AS uuid[])) "
                "  AND mapping_date = :mapping_date "
                "ORDER BY instrument_id, broker"
            ),
            {
                "instrument_identifiers": instrument_identifiers,
                "mapping_date": mapping_date,
            },
        ).all()

    handles_by_instrument = {}
    for row in rows:
        handles_by_instrument.setdefault(str(row.instrument_id), {})[row.broker] = self.handle_from_row(row)
    return handles_by_instrument

broker_tokens_on_date 

broker_tokens_on_date(mapping_date, broker)

Every token one broker maps on one date, which is what a whole-broker warm has to resolve.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required
broker str

The broker name, for example "zerodha".

 required

Returns:

Type Description
list[str]

The broker's tokens on that date.

Example usage

From example_1_point_reads_for_a_few_instruments.py
print(f'broker_tokens_on_date: {self.tier.broker_tokens_on_date(self.mapping_date, "zerodha")}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328
1329
1330
1331
1332
1333
1334
def broker_tokens_on_date(self, mapping_date, broker):
    """
    Every token one broker maps on one date, which is what a whole-broker warm has to resolve.

    Args:
        mapping_date (datetime.date): The mapping date to read.
        broker (str): The broker name, for example "zerodha".

    Returns:
        list[str]: The broker's tokens on that date.
    """
    with self.engine.connect() as connection:
        rows = connection.execute(
            text(
                f"SELECT DISTINCT broker_token FROM {tables.BROKER_MAPPINGS} "
                "WHERE broker = :broker AND mapping_date = :mapping_date"
            ),
            {
                "broker": broker,
                "mapping_date": mapping_date,
            },
        ).all()

    tokens = []
    for row in rows:
        tokens.append(row.broker_token)
    return tokens

stream_identities 

stream_identities(mapping_date)

Yield the identity of every instrument mapped on the date, one at a time.

The identity is produced once per instrument rather than once per token, because several brokers' tokens point at one instrument and repeating it per token would about double the work.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required

Yields:

Type Description
dict

One identity dict.

Example usage

From example_2_streaming_a_whole_day.py
self.tier.stream_identities(self.mapping_date)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
1350
1351
1352
1353
1354
1355
1356
1357
1358
1359
def stream_identities(self, mapping_date):
    """
    Yield the identity of every instrument mapped on the date, one at a time.

    The identity is produced once per instrument rather than once per token, because several brokers' tokens point at one instrument and repeating it per token would about double the work.

    Args:
        mapping_date (datetime.date): The mapping date to read.

    Yields:
        dict: One identity dict.
    """
    statement = text(
        "SELECT DISTINCT ON (m.instrument_id) "
        "  m.instrument_id, m.exchange, m.segment, m.shape, m.symbol, m.underlying_symbol, "
        "  m.expiry_date, m.strike_price, m.option_type "
        f"FROM {tables.BROKER_MAPPINGS} b "
        f"JOIN {tables.MASTER} m ON m.instrument_id = b.instrument_id "
        "WHERE b.mapping_date = :mapping_date "
        "ORDER BY m.instrument_id"
    )
    with self.streaming_connection() as connection:
        for row in connection.execute(statement, {"mapping_date": mapping_date}):
            yield self.identity_from_row(row, mapping_date)

stream_broker_tokens 

stream_broker_tokens(mapping_date, broker)

Yield one broker's token and instrument id pairs for the date, ordered so one token's rows arrive together.

The rows arrive in the same tie-break order the point look-up uses, so the first identity stored under a token is the answer an unfiltered look-up gets.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required
broker str

The broker name, for example "zerodha".

 required

Yields:

Type Description
tuple

The broker token as text and the instrument id as text.

Example usage

From example_2_streaming_a_whole_day.py
self.tier.stream_broker_tokens(self.mapping_date, 'zerodha')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1361
1362
1363
1364
1365
1366
1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
1377
1378
1379
1380
1381
1382
1383
1384
1385
1386
1387
1388
1389
1390
def stream_broker_tokens(self, mapping_date, broker):
    """
    Yield one broker's token and instrument id pairs for the date, ordered so one token's rows arrive together.

    The rows arrive in the same tie-break order the point look-up uses, so the first identity stored under a token is the answer an unfiltered look-up gets.

    Args:
        mapping_date (datetime.date): The mapping date to read.
        broker (str): The broker name, for example "zerodha".

    Yields:
        tuple: The broker token as text and the instrument id as text.
    """
    statement = text(
        "SELECT b.broker_token, m.instrument_id "
        f"FROM {tables.BROKER_MAPPINGS} b "
        f"JOIN {tables.MASTER} m ON m.instrument_id = b.instrument_id "
        "WHERE b.mapping_date = :mapping_date AND b.broker = :broker "
        "ORDER BY b.broker_token ASC, m.symbol ASC, m.instrument_id ASC"
    )
    with self.streaming_connection() as connection:
        rows = connection.execute(
            statement,
            {
                "mapping_date": mapping_date,
                "broker": broker,
            },
        )
        for row in rows:
            yield row.broker_token, str(row.instrument_id)

stream_order_handles 

stream_order_handles(mapping_date)

Yield every instrument's order handles for the date, ordered so one instrument's rows arrive together.

The same four columns the point look-up reads, so a warmed hash and a lazily filled one hold the same shape.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required

Yields:

Type Description
tuple

The instrument id as text, the broker name, and that broker's handle dict.

Example usage

From example_2_streaming_a_whole_day.py
self.tier.stream_order_handles(self.mapping_date)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1392
1393
1394
1395
1396
1397
1398
1399
1400
1401
1402
1403
1404
1405
1406
1407
1408
1409
1410
1411
1412
def stream_order_handles(self, mapping_date):
    """
    Yield every instrument's order handles for the date, ordered so one instrument's rows arrive together.

    The same four columns the point look-up reads, so a warmed hash and a lazily filled one hold the same shape.

    Args:
        mapping_date (datetime.date): The mapping date to read.

    Yields:
        tuple: The instrument id as text, the broker name, and that broker's handle dict.
    """
    statement = text(
        "SELECT instrument_id, broker, broker_token, order_symbol, lot_size, tick_size "
        f"FROM {tables.BROKER_MAPPINGS} "
        "WHERE mapping_date = :mapping_date "
        "ORDER BY instrument_id ASC, broker ASC"
    )
    with self.streaming_connection() as connection:
        for row in connection.execute(statement, {"mapping_date": mapping_date}):
            yield str(row.instrument_id), row.broker, self.handle_from_row(row)

stream_contract_sizes 

stream_contract_sizes(mapping_date)

Yield every contract size decision for the date.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required

Yields:

Type Description
tuple

The instrument id as text, the units per lot, the status and whether the contract is tradeable.

Example usage

From example_2_streaming_a_whole_day.py
self.tier.stream_contract_sizes(self.mapping_date)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1414
1415
1416
1417
1418
1419
1420
1421
1422
1423
1424
1425
1426
1427
1428
1429
1430
1431
def stream_contract_sizes(self, mapping_date):
    """
    Yield every contract size decision for the date.

    Args:
        mapping_date (datetime.date): The mapping date to read.

    Yields:
        tuple: The instrument id as text, the units per lot, the status and whether the contract is tradeable.
    """
    statement = text(
        "SELECT instrument_id, units_per_lot, status, tradeable "
        f"FROM {tables.CONTRACT_SIZES} "
        "WHERE mapping_date = :mapping_date"
    )
    with self.streaming_connection() as connection:
        for row in connection.execute(statement, {"mapping_date": mapping_date}):
            yield str(row.instrument_id), row.units_per_lot, row.status, row.tradeable

stream_underlyings 

stream_underlyings(mapping_date)

Yield every resolved underlying for the date.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required

Yields:

Type Description
tuple

The derivative's instrument id and its underlying's instrument id, both as text.

Example usage

From example_2_streaming_a_whole_day.py
self.tier.stream_underlyings(self.mapping_date)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1433
1434
1435
1436
1437
1438
1439
1440
1441
1442
1443
1444
1445
1446
1447
1448
1449
1450
def stream_underlyings(self, mapping_date):
    """
    Yield every resolved underlying for the date.

    Args:
        mapping_date (datetime.date): The mapping date to read.

    Yields:
        tuple: The derivative's instrument id and its underlying's instrument id, both as text.
    """
    statement = text(
        "SELECT instrument_id, underlying_instrument_id "
        f"FROM {tables.UNDERLYINGS} "
        "WHERE mapping_date = :mapping_date AND underlying_instrument_id IS NOT NULL"
    )
    with self.streaming_connection() as connection:
        for row in connection.execute(statement, {"mapping_date": mapping_date}):
            yield str(row.instrument_id), str(row.underlying_instrument_id)

read_underlyings 

read_underlyings(mapping_date, instrument_identifiers)

Read instruments' resolved underlyings straight out of unified.underlyings.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required
instrument_identifiers list[str]

The instrument ids to read.

 required

Returns:

Type Description
dict

Mapping of instrument id to its underlying's instrument id, both as text. Instruments with no resolved underlying are absent, and so is everything when the table does not exist yet.

Example usage

From example_1_point_reads_for_a_few_instruments.py
print(f'read_underlyings: {self.tier.read_underlyings(self.mapping_date, future_only)}')
From example_1_point_reads_for_a_few_instruments.py
print(f'read_underlyings of nothing: {self.tier.read_underlyings(self.mapping_date, [])}')
From example_2_streaming_a_whole_day.py
print(f'read_underlyings before the table exists: {fresh_tier.read_underlyings(self.mapping_date, wanted)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1452
1453
1454
1455
1456
1457
1458
1459
1460
1461
1462
1463
1464
1465
1466
1467
1468
1469
1470
1471
1472
1473
1474
1475
1476
1477
1478
1479
1480
1481
1482
def read_underlyings(self, mapping_date, instrument_identifiers):
    """
    Read instruments' resolved underlyings straight out of unified.underlyings.

    Args:
        mapping_date (datetime.date): The mapping date to read.
        instrument_identifiers (list[str]): The instrument ids to read.

    Returns:
        dict: Mapping of instrument id to its underlying's instrument id, both as text. Instruments with no resolved underlying are absent, and so is everything when the table does not exist yet.
    """
    if not instrument_identifiers:
        return {}
    statement = text(
        "SELECT instrument_id, underlying_instrument_id "
        f"FROM {tables.UNDERLYINGS} "
        "WHERE mapping_date = :mapping_date AND underlying_instrument_id IS NOT NULL "
        "AND instrument_id = ANY(CAST(:instrument_ids AS uuid[]))"
    )
    parameters = {
        "mapping_date": mapping_date,
        "instrument_ids": list(instrument_identifiers),
    }
    underlyings = {}
    try:
        with self.engine.connect() as connection:
            for row in connection.execute(statement, parameters):
                underlyings[str(row.instrument_id)] = str(row.underlying_instrument_id)
    except ProgrammingError:
        return {}
    return underlyings

stream_additional_attributes 

stream_additional_attributes(mapping_date)

Yield every instrument's additional broker attributes for the date, ordered so one instrument's rows arrive together.

Rows whose attributes are null are left out, because a broker that publishes nothing beyond the handle has nothing to say here and an entry of sixteen nulls is not worth the bytes.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required

Yields:

Type Description
tuple

The instrument id as text, the broker name, and that broker's attributes dict.

Example usage

From example_2_streaming_a_whole_day.py
self.tier.stream_additional_attributes(self.mapping_date)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1484
1485
1486
1487
1488
1489
1490
1491
1492
1493
1494
1495
1496
1497
1498
1499
1500
1501
1502
1503
1504
def stream_additional_attributes(self, mapping_date):
    """
    Yield every instrument's additional broker attributes for the date, ordered so one instrument's rows arrive together.

    Rows whose attributes are null are left out, because a broker that publishes nothing beyond the handle has nothing to say here and an entry of sixteen nulls is not worth the bytes.

    Args:
        mapping_date (datetime.date): The mapping date to read.

    Yields:
        tuple: The instrument id as text, the broker name, and that broker's attributes dict.
    """
    statement = text(
        "SELECT instrument_id, broker, attributes "
        f"FROM {tables.BROKER_MAPPINGS} "
        "WHERE mapping_date = :mapping_date AND attributes IS NOT NULL "
        "ORDER BY instrument_id ASC, broker ASC"
    )
    with self.streaming_connection() as connection:
        for row in connection.execute(statement, {"mapping_date": mapping_date}):
            yield str(row.instrument_id), row.broker, row.attributes

read_additional_attributes 

read_additional_attributes(mapping_date, instrument_identifiers)

Read instruments' additional broker attributes straight from the mapping table.

This is the answer for a past date and for a cache that has not been warmed, the two cases the Redis tier cannot serve.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required
instrument_identifiers list[str]

The instrument ids to read.

 required

Returns:

Type Description
dict

Mapping of instrument id to a dict of broker name to that broker's attributes. Instruments with no attributes on the date are absent.

Example usage

From example_1_point_reads_for_a_few_instruments.py
print(f'read_additional_attributes: {self.tier.read_additional_attributes(self.mapping_date, infy_only)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1506
1507
1508
1509
1510
1511
1512
1513
1514
1515
1516
1517
1518
1519
1520
1521
1522
1523
1524
1525
1526
1527
1528
1529
1530
1531
1532
1533
1534
1535
1536
1537
1538
1539
1540
def read_additional_attributes(self, mapping_date, instrument_identifiers):
    """
    Read instruments' additional broker attributes straight from the mapping table.

    This is the answer for a past date and for a cache that has not been warmed, the two cases the Redis tier cannot serve.

    Args:
        mapping_date (datetime.date): The mapping date to read.
        instrument_identifiers (list[str]): The instrument ids to read.

    Returns:
        dict: Mapping of instrument id to a dict of broker name to that broker's attributes. Instruments with no attributes on the date are absent.
    """
    if not instrument_identifiers:
        return {}

    statement = text(
        "SELECT instrument_id, broker, attributes "
        f"FROM {tables.BROKER_MAPPINGS} "
        "WHERE mapping_date = :mapping_date AND attributes IS NOT NULL "
        "  AND instrument_id = ANY(CAST(:instrument_identifiers AS uuid[])) "
        "ORDER BY instrument_id ASC, broker ASC"
    )
    attributes_by_instrument = {}
    with self.engine.connect() as connection:
        rows = connection.execute(statement, {
            "mapping_date": mapping_date,
            "instrument_identifiers": list(instrument_identifiers),
        }).all()
    for row in rows:
        instrument_identifier = str(row.instrument_id)
        if instrument_identifier not in attributes_by_instrument:
            attributes_by_instrument[instrument_identifier] = {}
        attributes_by_instrument[instrument_identifier][row.broker] = row.attributes
    return attributes_by_instrument

stream_catalogue 

stream_catalogue(mapping_date)

Yield the identity and seen dates of every instrument mapped on the date, one at a time.

The same instruments as stream_identities, with the two columns a catalogue needs beyond the identity.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required

Yields:

Type Description
tuple

The identity dict, its first seen date and its last seen date.

Example usage

From example_2_streaming_a_whole_day.py
self.tier.stream_catalogue(self.mapping_date)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1542
1543
1544
1545
1546
1547
1548
1549
1550
1551
1552
1553
1554
1555
1556
1557
1558
1559
1560
1561
1562
1563
1564
1565
def stream_catalogue(self, mapping_date):
    """
    Yield the identity and seen dates of every instrument mapped on the date, one at a time.

    The same instruments as stream_identities, with the two columns a catalogue needs beyond the identity.

    Args:
        mapping_date (datetime.date): The mapping date to read.

    Yields:
        tuple: The identity dict, its first seen date and its last seen date.
    """
    statement = text(
        "SELECT DISTINCT ON (m.instrument_id) "
        "  m.instrument_id, m.exchange, m.segment, m.shape, m.symbol, m.underlying_symbol, "
        "  m.expiry_date, m.strike_price, m.option_type, m.first_seen_date, m.last_seen_date "
        f"FROM {tables.BROKER_MAPPINGS} b "
        f"JOIN {tables.MASTER} m ON m.instrument_id = b.instrument_id "
        "WHERE b.mapping_date = :mapping_date "
        "ORDER BY m.instrument_id"
    )
    with self.streaming_connection() as connection:
        for row in connection.execute(statement, {"mapping_date": mapping_date}):
            yield self.identity_from_row(row, mapping_date), row.first_seen_date, row.last_seen_date

streaming_connection 

streaming_connection()

A connection that streams its results rather than buffering the whole answer.

A warm reads the entire mapping, which costs 829 megabytes to hold in one process. Streaming is what keeps the command that exists to spare everyone that cost from paying it itself.

Returns:

Type Description
Connection

A connection with streaming execution options set.

Example usage

From example_2_streaming_a_whole_day.py
self.tier.streaming_connection()
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1567
1568
1569
1570
1571
1572
1573
1574
1575
1576
1577
1578
1579
def streaming_connection(self):
    """
    A connection that streams its results rather than buffering the whole answer.

    A warm reads the entire mapping, which costs 829 megabytes to hold in one process. Streaming is what keeps the command that exists to spare everyone that cost from paying it itself.

    Returns:
        sqlalchemy.engine.Connection: A connection with streaming execution options set.
    """
    return self.engine.connect().execution_options(
        stream_results=True,
        max_row_buffer=self.STREAM_ROW_BUFFER,
    )

identity_from_row 

identity_from_row(row, mapping_date)

Build one identity dictionary from a database row.

Parameters:

Name Type Description Default
row Row

A row carrying the nine identity columns of unified.instruments.

 required
mapping_date date

The mapping date to stamp on the identity.

 required

Returns:

Type Description
dict

The identity, with its instrument id as text.

Example usage

From example_1_point_reads_for_a_few_instruments.py
print(f'identity_from_row: {self.tier.identity_from_row(row, self.mapping_date)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1581
1582
1583
1584
1585
1586
1587
1588
1589
1590
1591
1592
1593
1594
1595
1596
1597
1598
1599
1600
1601
1602
1603
def identity_from_row(self, row, mapping_date):
    """
    Build one identity dictionary from a database row.

    Args:
        row (sqlalchemy.engine.Row): A row carrying the nine identity columns of unified.instruments.
        mapping_date (datetime.date): The mapping date to stamp on the identity.

    Returns:
        dict: The identity, with its instrument id as text.
    """
    return {
        "instrument_id": str(row.instrument_id),
        "exchange": row.exchange,
        "segment": row.segment,
        "shape": row.shape,
        "symbol": row.symbol,
        "underlying_symbol": row.underlying_symbol,
        "expiry_date": row.expiry_date,
        "strike_price": row.strike_price,
        "option_type": row.option_type,
        "mapping_date": mapping_date,
    }

handle_from_row 

handle_from_row(row)

Build one broker's order handle from a database row.

The lot and tick sizes are kept as text rather than as decimals, because the handle is stored in Redis as JSON and a decimal that survives the round trip as a float would be a price-sized rounding error in a quantity.

Parameters:

Name Type Description Default
row Row

A row carrying broker_token, order_symbol, lot_size and tick_size.

 required

Returns:

Type Description
dict

The handle, with keys "broker_token", "order_symbol", "lot_size" and "tick_size".

Example usage

From example_1_point_reads_for_a_few_instruments.py
print(f'handle_from_row with no tick size: {self.tier.handle_from_row(row)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1605
1606
1607
1608
1609
1610
1611
1612
1613
1614
1615
1616
1617
1618
1619
1620
1621
1622
def handle_from_row(self, row):
    """
    Build one broker's order handle from a database row.

    The lot and tick sizes are kept as text rather than as decimals, because the handle is stored in Redis as JSON and a decimal that survives the round trip as a float would be a price-sized rounding error in a quantity.

    Args:
        row (sqlalchemy.engine.Row): A row carrying broker_token, order_symbol, lot_size and tick_size.

    Returns:
        dict: The handle, with keys "broker_token", "order_symbol", "lot_size" and "tick_size".
    """
    return {
        "broker_token": row.broker_token,
        "order_symbol": row.order_symbol,
        "lot_size": None if row.lot_size is None else str(row.lot_size),
        "tick_size": None if row.tick_size is None else str(row.tick_size),
    }

ThreeTierLookup 

ThreeTierLookup(redis_tier, postgres_tier, statistics)

The walk from this process's memory to Redis to Postgres, shared by the three questions the cache answers.

Each tier fills in the ones above it, so the first look-up of a key costs a Redis round trip or a Postgres query and every look-up after it costs a dictionary read. Subclasses supply the two reads and the one write; everything else about the walk, including the counting, is here.

Attributes:

Name Type Description
redis_tier MappingRedisTier

The Redis hashes this lookup reads and writes.
postgres_tier MappingPostgresTier

The statements this lookup falls through to.
statistics dict

The counters shared with every other lookup on the same cache, so the tally is per process rather than per question.

Build the lookup around the two lower tiers and the shared counters.

Parameters:

Name Type Description Default
redis_tier MappingRedisTier

The Redis hashes to read and write.

 required
postgres_tier MappingPostgresTier

The statements to fall through to.

 required
statistics dict

The counters to count into.

 required

Returns:

Type Description
None

This function returns nothing.

Example programs

examples/stock_brokers/instruments/mapping/utilities/cache/ThreeTierLookup/example_1_a_lot_size_lookup.py
"""Builds a small lookup on top of `ThreeTierLookup` and follows one key from Postgres into Redis and into process memory.

`ThreeTierLookup` holds the walk every mapping cache question shares: look in this process's own dictionary, then in Redis, then in Postgres, and copy each answer into the tiers above the one that had it. A subclass supplies only three methods, `read_from_redis`, `read_from_postgres` and `write_to_redis`. The three lookups in `cache.py` are such subclasses; this program writes a fourth, which answers an instrument id with its lot size, so the base class's own behaviour can be seen without the real tiers' detail.

The "Redis" and "Postgres" tiers here are two small stand-in classes holding dictionaries, because the base class never looks inside its tiers and only the subclass talks to them. Notice how the shared `statistics` counters move: the first `resolve` is answered by Postgres, the second by this process, and a second lookup, standing in for another process, is answered by Redis because the first one wrote its answer there. `remember`, `known`, `entry` and `reset` work on the process tier alone and never touch the other two.

Run it from the project root:

    python examples/stock_brokers/instruments/mapping/utilities/cache/ThreeTierLookup/example_1_a_lot_size_lookup.py
"""

import datetime

from stock_brokers.instruments.mapping.utilities.cache import (
    ThreeTierLookup,
)


class DictionaryTier:
    """A stand-in for a Redis or Postgres tier that holds lot sizes in a dictionary.

    Attributes:
        lot_sizes (dict): Each instrument id's lot size.
        reads (int): How many times the tier was read.
    """

    def __init__(self, lot_sizes):
        """Builds the tier.

        Args:
            lot_sizes (dict): Each instrument id's lot size.

        Returns:
            None: This method returns nothing.
        """
        self.lot_sizes = lot_sizes
        self.reads = 0

    def read(self, keys):
        """Reads the lot sizes of the keys this tier holds.

        Args:
            keys (list): The instrument ids to read.

        Returns:
            dict: Each held instrument id to an entry with its lot size.
        """
        self.reads += 1
        found = {}
        for key in keys:
            if key in self.lot_sizes:
                found[key] = {
                    'lot_size': self.lot_sizes[key],
                }
        return found

    def write(self, entries):
        """Stores entries' lot sizes.

        Args:
            entries (dict): Each instrument id to an entry with its lot size.

        Returns:
            bool: Always True.
        """
        for key, entry in entries.items():
            self.lot_sizes[key] = entry['lot_size']
        return True


class LotSizeLookup(ThreeTierLookup):
    """Instrument id to that instrument's lot size, through the three tiers."""

    def read_from_redis(self, mapping_date, keys):
        """Reads lot sizes out of the Redis stand-in.

        Args:
            mapping_date (datetime.date): The mapping date to read.
            keys (list): The instrument ids to read.

        Returns:
            dict: Each held instrument id to its entry.
        """
        return self.redis_tier.read(keys)

    def read_from_postgres(self, mapping_date, keys):
        """Reads lot sizes out of the Postgres stand-in.

        Args:
            mapping_date (datetime.date): The mapping date to read.
            keys (list): The instrument ids to read.

        Returns:
            dict: Each held instrument id to its entry.
        """
        return self.postgres_tier.read(keys)

    def write_to_redis(self, mapping_date, entries):
        """Writes lot sizes into the Redis stand-in.

        Args:
            mapping_date (datetime.date): The mapping date the entries belong to.
            entries (dict): Each instrument id to its entry.

        Returns:
            bool: True when the write went through.
        """
        return self.redis_tier.write(entries)


class LotSizeLookupExample:
    """Resolves lot sizes through a first and a second lookup sharing one Redis stand-in.

    Attributes:
        mapping_date (datetime.date): The mapping date asked about.
        redis_tier (DictionaryTier): The Redis stand-in, empty at first.
        postgres_tier (DictionaryTier): The Postgres stand-in, holding every lot size.
        statistics (dict): The counters the lookups share.
        lookup (LotSizeLookup): The first process's lookup.
    """

    def __init__(self):
        """Builds the tiers, the counters and the first lookup.

        Returns:
            None: This method returns nothing.
        """
        self.mapping_date = datetime.date(2026, 9, 30)
        self.redis_tier = DictionaryTier({})
        self.postgres_tier = DictionaryTier({
            'nifty-future': '75',
            'banknifty-future': '35',
            'crudeoil-future': '100',
        })
        self.statistics = {
            'process_hits': 0,
            'redis_hits': 0,
            'postgres_hits': 0,
            'misses': 0,
        }
        self.lookup = LotSizeLookup(self.redis_tier, self.postgres_tier, self.statistics)

    def run(self):
        """Resolves the same keys cold, warm and from a second lookup, then uses the process tier directly.

        Returns:
            None: This method returns nothing.
        """
        keys = [
            'nifty-future',
            'banknifty-future',
        ]
        print(f'First resolve: {self.lookup.resolve(self.mapping_date, keys)}')
        print(f'  counters: {self.statistics}')
        print(f'Second resolve: {self.lookup.resolve(self.mapping_date, keys)}')
        print(f'  counters: {self.statistics}')
        print(f'Redis now holds: {self.redis_tier.lot_sizes}')
        second_process = LotSizeLookup(self.redis_tier, self.postgres_tier, self.statistics)
        print(f'Second process resolves: {second_process.resolve(self.mapping_date, keys)}')
        print(f'  counters: {self.statistics}')
        print(f'Postgres was read {self.postgres_tier.reads} time')
        print(f'Known before remember: {self.lookup.known("crudeoil-future")}')
        crude_entry = {
            'lot_size': '100',
        }
        self.lookup.remember('crudeoil-future', crude_entry)
        print(f'Known after remember: {self.lookup.known("crudeoil-future")}')
        print(f'Entry held for it: {self.lookup.entry("crudeoil-future")}')
        print(f'Entry for a key never seen: {self.lookup.entry("goldm-future")}')
        print(f'What a miss is remembered as: {self.lookup.empty_entry()}')
        crude_only = [
            'crudeoil-future',
        ]
        print(f'Reading Redis directly: {self.lookup.read_from_redis(self.mapping_date, crude_only)}')
        print(f'Reading Postgres directly: {self.lookup.read_from_postgres(self.mapping_date, crude_only)}')
        crude_entries = {
            'crudeoil-future': crude_entry,
        }
        print(f'Writing Redis directly: {self.lookup.write_to_redis(self.mapping_date, crude_entries)}')
        self.lookup.reset()
        print(f'Known after reset: {self.lookup.known("nifty-future")}')


if __name__ == '__main__':
    LotSizeLookupExample().run()

It prints:

First resolve: {'nifty-future': {'lot_size': '75'}, 'banknifty-future': {'lot_size': '35'}}
  counters: {'process_hits': 0, 'redis_hits': 0, 'postgres_hits': 2, 'misses': 0}
Second resolve: {'nifty-future': {'lot_size': '75'}, 'banknifty-future': {'lot_size': '35'}}
  counters: {'process_hits': 2, 'redis_hits': 0, 'postgres_hits': 2, 'misses': 0}
Redis now holds: {'nifty-future': '75', 'banknifty-future': '35'}
Second process resolves: {'nifty-future': {'lot_size': '75'}, 'banknifty-future': {'lot_size': '35'}}
  counters: {'process_hits': 2, 'redis_hits': 2, 'postgres_hits': 2, 'misses': 0}
Postgres was read 1 time
Known before remember: False
Known after remember: True
Entry held for it: {'lot_size': '100'}
Entry for a key never seen: None
What a miss is remembered as: None
Reading Redis directly: {}
Reading Postgres directly: {'crudeoil-future': {'lot_size': '100'}}
Writing Redis directly: True
Known after reset: False
examples/stock_brokers/instruments/mapping/utilities/cache/ThreeTierLookup/example_2_remembering_misses.py
"""Shows what the base class does on its own, and how `empty_entry` decides whether a miss is asked about again.

`ThreeTierLookup` is meant to be subclassed. Used directly, its three tier methods raise `NotImplementedError` naming the class, while an empty batch of keys is answered without reaching any tier at all.

The more interesting choice a subclass makes is `empty_entry`. When Postgres has no answer for a key, the walk asks `empty_entry()` what to remember. The base answer, None, remembers nothing, so the next look-up of that key goes all the way to Postgres again; the cache's identity lookup works this way, because an unknown instrument id is a caller's mistake that tomorrow's master may fix. A subclass that returns an empty value instead makes the miss a fact for the rest of the day; the cache's token lookup works this way, because a token a broker does not carry is asked about on every tick.

This program builds two small subclasses that differ only in `empty_entry` and asks each about the same unknown key three times, against a stand-in Postgres that counts its reads. Notice that the first asks Postgres three times and the second once.

Run it from the project root:

    python examples/stock_brokers/instruments/mapping/utilities/cache/ThreeTierLookup/example_2_remembering_misses.py
"""

import datetime

from stock_brokers.instruments.mapping.utilities.cache import (
    ThreeTierLookup,
)


class CountingEmptyTier:
    """A stand-in tier that holds nothing and counts how often it is read.

    Attributes:
        reads (int): How many times the tier was read.
    """

    def __init__(self):
        """Builds the tier with no reads counted.

        Returns:
            None: This method returns nothing.
        """
        self.reads = 0

    def read(self, keys):
        """Counts one read and finds nothing.

        Args:
            keys (list): The keys asked about.

        Returns:
            dict: Always empty.
        """
        self.reads += 1
        return {}


class ForgettingLookup(ThreeTierLookup):
    """A lookup that keeps the base class's `empty_entry`, so it never remembers a miss."""

    def read_from_redis(self, mapping_date, keys):
        """Reads the Redis stand-in.

        Args:
            mapping_date (datetime.date): The mapping date to read.
            keys (list): The keys to read.

        Returns:
            dict: What the stand-in held.
        """
        return self.redis_tier.read(keys)

    def read_from_postgres(self, mapping_date, keys):
        """Reads the Postgres stand-in.

        Args:
            mapping_date (datetime.date): The mapping date to read.
            keys (list): The keys to read.

        Returns:
            dict: What the stand-in held.
        """
        return self.postgres_tier.read(keys)

    def write_to_redis(self, mapping_date, entries):
        """Accepts entries without storing them.

        Args:
            mapping_date (datetime.date): The mapping date the entries belong to.
            entries (dict): The entries to write.

        Returns:
            bool: Always True.
        """
        return True


class RememberingLookup(ForgettingLookup):
    """A lookup that remembers a miss as an empty list for the rest of the day."""

    def empty_entry(self):
        """What to remember for a key nothing maps.

        Returns:
            list: A new empty list.
        """
        return []


class RememberingMissesExample:
    """Calls the base class directly, then compares a forgetting and a remembering lookup.

    Attributes:
        mapping_date (datetime.date): The mapping date asked about.
        statistics (dict): The counters the lookups count into.
    """

    def __init__(self):
        """Builds the date and the counters.

        Returns:
            None: This method returns nothing.
        """
        self.mapping_date = datetime.date(2026, 9, 30)
        self.statistics = {
            'process_hits': 0,
            'redis_hits': 0,
            'postgres_hits': 0,
            'misses': 0,
        }

    def show_base_class(self):
        """Calls each method of the base class itself and prints what happens.

        Returns:
            None: This method returns nothing.
        """
        base = ThreeTierLookup(None, None, self.statistics)
        print(f'Base resolve of no keys: {base.resolve(self.mapping_date, [])}')
        print(f'Base empty_entry: {base.empty_entry()}')
        keys = [
            '999999',
        ]
        try:
            base.resolve(self.mapping_date, keys)
        except NotImplementedError as error:
            print(f'Base resolve of a key: NotImplementedError: {error}')
        try:
            base.read_from_redis(self.mapping_date, keys)
        except NotImplementedError as error:
            print(f'Base read_from_redis: NotImplementedError: {error}')
        try:
            base.read_from_postgres(self.mapping_date, keys)
        except NotImplementedError as error:
            print(f'Base read_from_postgres: NotImplementedError: {error}')
        try:
            base.write_to_redis(self.mapping_date, {})
        except NotImplementedError as error:
            print(f'Base write_to_redis: NotImplementedError: {error}')

    def ask_three_times(self, lookup, postgres_tier):
        """Asks one lookup about an unknown key three times and prints the result.

        Args:
            lookup (ThreeTierLookup): The lookup to ask.
            postgres_tier (CountingEmptyTier): The Postgres stand-in behind it.

        Returns:
            None: This method returns nothing.
        """
        keys = [
            '999999',
        ]
        answers = []
        for attempt in range(3):
            answers.append(lookup.resolve(self.mapping_date, keys))
        name = type(lookup).__name__
        print(f'{name}: answers {answers}, Postgres read {postgres_tier.reads} times, remembered: {lookup.known("999999")}')

    def run(self):
        """Shows the base class, then the two lookups side by side.

        Returns:
            None: This method returns nothing.
        """
        self.show_base_class()
        forgetting_postgres = CountingEmptyTier()
        forgetting = ForgettingLookup(CountingEmptyTier(), forgetting_postgres, self.statistics)
        self.ask_three_times(forgetting, forgetting_postgres)
        remembering_postgres = CountingEmptyTier()
        remembering = RememberingLookup(CountingEmptyTier(), remembering_postgres, self.statistics)
        self.ask_three_times(remembering, remembering_postgres)
        print(f'Counters: {self.statistics}')


if __name__ == '__main__':
    RememberingMissesExample().run()

It prints:

Base resolve of no keys: {}
Base empty_entry: None
Base resolve of a key: NotImplementedError: ThreeTierLookup must implement read_from_redis()
Base read_from_redis: NotImplementedError: ThreeTierLookup must implement read_from_redis()
Base read_from_postgres: NotImplementedError: ThreeTierLookup must implement read_from_postgres()
Base write_to_redis: NotImplementedError: ThreeTierLookup must implement write_to_redis()
ForgettingLookup: answers [{}, {}, {}], Postgres read 3 times, remembered: False
RememberingLookup: answers [{'999999': []}, {'999999': []}, {'999999': []}], Postgres read 1 times, remembered: True
Counters: {'process_hits': 2, 'redis_hits': 0, 'postgres_hits': 0, 'misses': 4}

Methods:

Name Description
resolve

Answer for a batch of keys, trying each tier in turn and filling in the ones above.
remember

Put one entry into this process's memory without going to any other tier.
known

Whether this process already holds an entry for one key.
entry

The entry this process holds for one key, without reaching any other tier.
reset

Drop everything this lookup holds in process memory.
empty_entry

What to remember for a key nothing maps, or None to remember nothing.
read_from_redis

Read a batch of keys out of Redis.
read_from_postgres

Read a batch of keys out of Postgres.
write_to_redis

Write entries resolved from Postgres back into Redis, so the next process does not repeat the query.
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1637
1638
1639
1640
1641
1642
1643
1644
1645
1646
1647
1648
1649
1650
1651
1652
1653
def __init__(self, redis_tier, postgres_tier, statistics):
    """
    Build the lookup around the two lower tiers and the shared counters.

    Args:
        redis_tier (MappingRedisTier): The Redis hashes to read and write.
        postgres_tier (MappingPostgresTier): The statements to fall through to.
        statistics (dict): The counters to count into.

    Returns:
        None: This function returns nothing.
    """
    self.redis_tier = redis_tier
    self.postgres_tier = postgres_tier
    self.statistics = statistics

    self._entries = {}

resolve 

resolve(mapping_date, keys)

Answer for a batch of keys, trying each tier in turn and filling in the ones above.

A key that nothing maps is remembered as an empty entry when the subclass says so, which stops a token nothing carries being asked about on every call. A subclass whose empty_entry is None does not remember misses at all.

Parameters:

Name Type Description Default
mapping_date date

The mapping date being answered for.

 required
keys list[str]

The keys to answer for.

 required

Returns:

Type Description
dict

Mapping of key to its entry, for every key that had one. Keys nothing maps are absent, unless the subclass remembers empty entries, in which case they carry one.

Example usage

From example_1_a_lot_size_lookup.py
print(f'First resolve: {self.lookup.resolve(self.mapping_date, keys)}')
From example_1_a_lot_size_lookup.py
print(f'Second resolve: {self.lookup.resolve(self.mapping_date, keys)}')
From example_1_a_lot_size_lookup.py
print(f'Second process resolves: {second_process.resolve(self.mapping_date, keys)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1655
1656
1657
1658
1659
1660
1661
1662
1663
1664
1665
1666
1667
1668
1669
1670
1671
1672
1673
1674
1675
1676
1677
1678
1679
1680
1681
1682
1683
1684
1685
1686
1687
1688
1689
1690
1691
1692
1693
1694
1695
1696
1697
1698
1699
1700
1701
1702
1703
1704
1705
1706
1707
1708
1709
def resolve(self, mapping_date, keys):
    """
    Answer for a batch of keys, trying each tier in turn and filling in the ones above.

    A key that nothing maps is remembered as an empty entry when the subclass says so, which stops a token nothing carries being asked about on every call. A subclass whose empty_entry is None does not remember misses at all.

    Args:
        mapping_date (datetime.date): The mapping date being answered for.
        keys (list[str]): The keys to answer for.

    Returns:
        dict: Mapping of key to its entry, for every key that had one. Keys nothing maps are absent, unless the subclass remembers empty entries, in which case they carry one.
    """
    if not keys:
        return {}

    resolved = {}
    outstanding = []
    for key in keys:
        entry = self._entries.get(key)
        if entry is None:
            outstanding.append(key)
            continue
        self.statistics["process_hits"] += 1
        resolved[key] = entry

    if outstanding:
        from_redis = self.read_from_redis(mapping_date, outstanding)
        still_outstanding = []
        for key in outstanding:
            if key not in from_redis:
                still_outstanding.append(key)
                continue
            self.statistics["redis_hits"] += 1
            self._entries[key] = from_redis[key]
            resolved[key] = from_redis[key]
        outstanding = still_outstanding

    if outstanding:
        from_postgres = self.read_from_postgres(mapping_date, outstanding)
        for key in outstanding:
            entry = from_postgres.get(key)
            if entry is None:
                self.statistics["misses"] += 1
                empty = self.empty_entry()
                if empty is not None:
                    self._entries[key] = empty
                    resolved[key] = empty
                continue
            self.statistics["postgres_hits"] += 1
            self._entries[key] = entry
            resolved[key] = entry
        self.write_to_redis(mapping_date, from_postgres)

    return resolved

remember 

remember(key, entry)

Put one entry into this process's memory without going to any other tier.

This exists so that a lookup filling itself can also fill a sibling, which is how a token's candidate identities land in the identity lookup rather than being read again from Redis.

Parameters:

Name Type Description Default
key str

The key to remember the entry under.

 required
entry dict

The entry to remember.

 required

Returns:

Type Description
None

This function returns nothing.

Example usage

From example_1_a_lot_size_lookup.py
self.lookup.remember('crudeoil-future', crude_entry)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1711
1712
1713
1714
1715
1716
1717
1718
1719
1720
1721
1722
1723
1724
def remember(self, key, entry):
    """
    Put one entry into this process's memory without going to any other tier.

    This exists so that a lookup filling itself can also fill a sibling, which is how a token's candidate identities land in the identity lookup rather than being read again from Redis.

    Args:
        key (str): The key to remember the entry under.
        entry (dict): The entry to remember.

    Returns:
        None: This function returns nothing.
    """
    self._entries[key] = entry

known 

known(key)

Whether this process already holds an entry for one key.

Parameters:

Name Type Description Default
key str

The key to test.

 required

Returns:

Type Description
bool

True when the entry is in this process's memory.

Example usage

From example_1_a_lot_size_lookup.py
print(f'Known before remember: {self.lookup.known("crudeoil-future")}')
From example_1_a_lot_size_lookup.py
print(f'Known after remember: {self.lookup.known("crudeoil-future")}')
From example_1_a_lot_size_lookup.py
print(f'Known after reset: {self.lookup.known("nifty-future")}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1726
1727
1728
1729
1730
1731
1732
1733
1734
1735
1736
def known(self, key):
    """
    Whether this process already holds an entry for one key.

    Args:
        key (str): The key to test.

    Returns:
        bool: True when the entry is in this process's memory.
    """
    return key in self._entries

entry 

entry(key)

The entry this process holds for one key, without reaching any other tier.

The counterpart of remember, for a sibling lookup assembling an answer out of entries this one already has.

Parameters:

Name Type Description Default
key str

The key to read.

 required

Returns:

Type Description
dict | None

The entry, or None when this process does not hold one.

Example usage

From example_1_a_lot_size_lookup.py
print(f'Entry held for it: {self.lookup.entry("crudeoil-future")}')
From example_1_a_lot_size_lookup.py
print(f'Entry for a key never seen: {self.lookup.entry("goldm-future")}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1738
1739
1740
1741
1742
1743
1744
1745
1746
1747
1748
1749
1750
def entry(self, key):
    """
    The entry this process holds for one key, without reaching any other tier.

    The counterpart of remember, for a sibling lookup assembling an answer out of entries this one already has.

    Args:
        key (str): The key to read.

    Returns:
        dict | None: The entry, or None when this process does not hold one.
    """
    return self._entries.get(key)

reset 

reset()

Drop everything this lookup holds in process memory.

Returns:

Type Description
None

This function returns nothing.

Example usage

From example_1_a_lot_size_lookup.py
self.lookup.reset()
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1752
1753
1754
1755
1756
1757
1758
1759
def reset(self):
    """
    Drop everything this lookup holds in process memory.

    Returns:
        None: This function returns nothing.
    """
    self._entries.clear()

empty_entry 

empty_entry()

What to remember for a key nothing maps, or None to remember nothing.

A fresh object is returned on every call rather than a shared one, so that an entry a caller narrows in place cannot reach the next key that also missed.

Returns:

Type Description
None

The base class does not remember misses.

Example usage

From example_1_a_lot_size_lookup.py
print(f'What a miss is remembered as: {self.lookup.empty_entry()}')
From example_2_remembering_misses.py
print(f'Base empty_entry: {base.empty_entry()}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1761
1762
1763
1764
1765
1766
1767
1768
1769
1770
def empty_entry(self):
    """
    What to remember for a key nothing maps, or None to remember nothing.

    A fresh object is returned on every call rather than a shared one, so that an entry a caller narrows in place cannot reach the next key that also missed.

    Returns:
        None: The base class does not remember misses.
    """
    return None

read_from_redis 

read_from_redis(mapping_date, keys)

Read a batch of keys out of Redis.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required
keys list[str]

The keys to read.

 required

Returns:

Type Description
dict

Mapping of key to entry, for the keys Redis held.

Raises:

Type Description
NotImplementedError

Always, unless the subclass overrides this method.

Example usage

From example_1_a_lot_size_lookup.py
print(f'Reading Redis directly: {self.lookup.read_from_redis(self.mapping_date, crude_only)}')
From example_2_remembering_misses.py
base.read_from_redis(self.mapping_date, keys)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1772
1773
1774
1775
1776
1777
1778
1779
1780
1781
1782
1783
1784
1785
1786
def read_from_redis(self, mapping_date, keys):
    """
    Read a batch of keys out of Redis.

    Args:
        mapping_date (datetime.date): The mapping date to read.
        keys (list[str]): The keys to read.

    Returns:
        dict: Mapping of key to entry, for the keys Redis held.

    Raises:
        NotImplementedError: Always, unless the subclass overrides this method.
    """
    raise NotImplementedError(f"{type(self).__name__} must implement read_from_redis()")

read_from_postgres 

read_from_postgres(mapping_date, keys)

Read a batch of keys out of Postgres.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required
keys list[str]

The keys to read.

 required

Returns:

Type Description
dict

Mapping of key to entry, for the keys Postgres held.

Raises:

Type Description
NotImplementedError

Always, unless the subclass overrides this method.

Example usage

From example_1_a_lot_size_lookup.py
print(f'Reading Postgres directly: {self.lookup.read_from_postgres(self.mapping_date, crude_only)}')
From example_2_remembering_misses.py
base.read_from_postgres(self.mapping_date, keys)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1788
1789
1790
1791
1792
1793
1794
1795
1796
1797
1798
1799
1800
1801
1802
def read_from_postgres(self, mapping_date, keys):
    """
    Read a batch of keys out of Postgres.

    Args:
        mapping_date (datetime.date): The mapping date to read.
        keys (list[str]): The keys to read.

    Returns:
        dict: Mapping of key to entry, for the keys Postgres held.

    Raises:
        NotImplementedError: Always, unless the subclass overrides this method.
    """
    raise NotImplementedError(f"{type(self).__name__} must implement read_from_postgres()")

write_to_redis 

write_to_redis(mapping_date, entries)

Write entries resolved from Postgres back into Redis, so the next process does not repeat the query.

Parameters:

Name Type Description Default
mapping_date date

The mapping date the entries belong to.

 required
entries dict

Mapping of key to entry.

 required

Returns:

Type Description
bool

True when the write went through, False when Redis was not usable.

Raises:

Type Description
NotImplementedError

Always, unless the subclass overrides this method.

Example usage

From example_1_a_lot_size_lookup.py
print(f'Writing Redis directly: {self.lookup.write_to_redis(self.mapping_date, crude_entries)}')
From example_2_remembering_misses.py
base.write_to_redis(self.mapping_date, {})
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1804
1805
1806
1807
1808
1809
1810
1811
1812
1813
1814
1815
1816
1817
1818
def write_to_redis(self, mapping_date, entries):
    """
    Write entries resolved from Postgres back into Redis, so the next process does not repeat the query.

    Args:
        mapping_date (datetime.date): The mapping date the entries belong to.
        entries (dict): Mapping of key to entry.

    Returns:
        bool: True when the write went through, False when Redis was not usable.

    Raises:
        NotImplementedError: Always, unless the subclass overrides this method.
    """
    raise NotImplementedError(f"{type(self).__name__} must implement write_to_redis()")

InstrumentIdentityLookup 

InstrumentIdentityLookup(redis_tier, postgres_tier, statistics)

Bases: ThreeTierLookup

Instrument id to that instrument's identity.

This is the direction a caller holding an instrument id and about to place an order on it needs: the exchange, the segment and the shape decide every broker-native code the order is sent with. It is also the store the token lookup fills, since a token's candidates are identities and the two must not hold separate copies.

A miss is not remembered, because an instrument id the master does not carry is a caller's mistake rather than a fact about the mapping, and remembering it would keep answering a question whose answer may arrive with the next day's master.

Example programs

examples/stock_brokers/instruments/mapping/utilities/cache/InstrumentIdentityLookup/example_1_filling_each_tier_in_turn.py
"""Resolves instrument ids to identities through process memory, Redis and Postgres, and shows each tier filling the one above.

An `InstrumentIdentityLookup` answers the question an order path asks just before it places an order: given a unified instrument id, what are its exchange, segment, shape, symbol, expiry, strike and option type? It is one of the three subclasses of `ThreeTierLookup` in `cache.py`, and it reads the identity hash through `MappingRedisTier` and `unified.instruments` through `MappingPostgresTier`.

Two stand-ins replace the data stores. The Redis stand-in keeps hashes in dictionaries and runs a pipeline's commands when it is executed. The Postgres stand-in is an engine that holds fixed rows for the identity statement, keeps only the ids a statement asks for, and records every call. Both are real `MappingRedisTier` and `MappingPostgresTier` objects on top, so the encoding and the row conversion are the project's own.

Notice the counters. The first `resolve` is answered by Postgres and writes both identities into the Redis hash; the second is answered from this process's memory; and a second lookup, standing in for another process that shares the same Redis, is answered by Redis without a query. The dates and the strike come back as `datetime.date` and `decimal.Decimal` from every tier.

Run it from the project root:

    python examples/stock_brokers/instruments/mapping/utilities/cache/InstrumentIdentityLookup/example_1_filling_each_tier_in_turn.py
"""

import datetime
import decimal
import types
import uuid

from stock_brokers.instruments.mapping.utilities.cache import (
    InstrumentIdentityLookup,
    MappingPostgresTier,
    MappingRedisConnection,
    MappingRedisTier,
)


class StandInPipeline:
    """A stand-in for a Redis pipeline that queues commands and runs them on execute.

    Attributes:
        redis_client (StandInRedis): The stand-in client the commands run against.
        queued (list): The queued commands, as (method, arguments) pairs.
    """

    def __init__(self, redis_client):
        """Builds an empty pipeline.

        Args:
            redis_client (StandInRedis): The stand-in client the commands run against.

        Returns:
            None: This method returns nothing.
        """
        self.redis_client = redis_client
        self.queued = []

    def set(self, key, value):
        """Queues a string write.

        Args:
            key (str): The key to write.
            value (str): The value to store.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.set, (key, value)))

    def hset(self, key, mapping):
        """Queues a hash write.

        Args:
            key (str): The hash to write into.
            mapping (dict): The fields to write.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.hset, (key, mapping)))

    def hmget(self, key, fields):
        """Queues a read of several hash fields.

        Args:
            key (str): The hash to read.
            fields (list): The fields to read.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.hmget, (key, fields)))

    def expire(self, key, seconds):
        """Queues an expiry.

        Args:
            key (str): The key to expire.
            seconds (int): How long the key may live.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.expire, (key, seconds)))

    def execute(self):
        """Runs every queued command in order.

        Returns:
            list: Each command's reply, in the order queued.
        """
        replies = []
        for method, arguments in self.queued:
            replies.append(method(*arguments))
        self.queued = []
        return replies


class StandInRedis:
    """A stand-in for a Redis client that keeps strings and hashes in dictionaries.

    Attributes:
        values (dict): Each key's value: a string, or a dict for a hash.
        expiries (dict): Each key's expiry in seconds, as last set.
    """

    def __init__(self):
        """Builds an empty stand-in.

        Returns:
            None: This method returns nothing.
        """
        self.values = {}
        self.expiries = {}

    def pipeline(self, transaction=True):
        """Starts a pipeline.

        Args:
            transaction (bool): Ignored; the stand-in always runs commands in order.

        Returns:
            StandInPipeline: A new pipeline.
        """
        return StandInPipeline(self)

    def get(self, key):
        """Reads a string.

        Args:
            key (str): The key to read.

        Returns:
            str | None: The value, or None when the key is absent.
        """
        return self.values.get(key)

    def set(self, key, value):
        """Writes a string.

        Args:
            key (str): The key to write.
            value (str): The value to store.

        Returns:
            bool: Always True.
        """
        self.values[key] = value
        return True

    def exists(self, key):
        """Counts whether a key exists.

        Args:
            key (str): The key to test.

        Returns:
            int: 1 when the key exists, otherwise 0.
        """
        if key in self.values:
            return 1
        return 0

    def expire(self, key, seconds):
        """Records a key's expiry.

        Args:
            key (str): The key to expire.
            seconds (int): How long the key may live.

        Returns:
            bool: Always True.
        """
        self.expiries[key] = seconds
        return True

    def hset(self, key, mapping):
        """Writes fields into a hash.

        Args:
            key (str): The hash to write into.
            mapping (dict): The fields to write.

        Returns:
            int: The number of fields written.
        """
        stored = self.values.setdefault(key, {})
        stored.update(mapping)
        return len(mapping)

    def hmget(self, key, fields):
        """Reads several fields of a hash.

        Args:
            key (str): The hash to read.
            fields (list): The fields to read.

        Returns:
            list: Each field's value, or None where the field is absent.
        """
        stored = self.values.get(key, {})
        replies = []
        for field in fields:
            replies.append(stored.get(field))
        return replies

    def hgetall(self, key):
        """Reads a whole hash.

        Args:
            key (str): The hash to read.

        Returns:
            dict: The hash's fields, empty when the key is absent.
        """
        return dict(self.values.get(key, {}))


class StandInResult:
    """A stand-in for a SQLAlchemy result holding rows already fetched.

    Attributes:
        rows (list): The rows, in order.
    """

    def __init__(self, rows):
        """Holds the rows.

        Args:
            rows (list): The rows, in order.

        Returns:
            None: This method returns nothing.
        """
        self.rows = rows

    def __iter__(self):
        """Iterates over the rows.

        Returns:
            iterator: An iterator over the rows.
        """
        return iter(self.rows)

    def all(self):
        """Returns every row.

        Returns:
            list: The rows.
        """
        return list(self.rows)

    def scalar(self):
        """Returns the first row's single value.

        Returns:
            object: The first row, which for a scalar answer is the value itself, or None when there is no row.
        """
        if not self.rows:
            return None
        return self.rows[0]


class StandInConnection:
    """A stand-in for a SQLAlchemy connection that asks its engine for each statement's rows.

    Attributes:
        engine (StandInEngine): The engine holding the answers.
    """

    def __init__(self, engine):
        """Builds the connection.

        Args:
            engine (StandInEngine): The engine holding the answers.

        Returns:
            None: This method returns nothing.
        """
        self.engine = engine

    def __enter__(self):
        """Enters a `with` block.

        Returns:
            StandInConnection: This connection.
        """
        return self

    def __exit__(self, exception_type, exception, traceback):
        """Leaves a `with` block without suppressing any exception.

        Args:
            exception_type (type | None): The exception's class, if one was raised.
            exception (BaseException | None): The exception, if one was raised.
            traceback (types.TracebackType | None): The traceback, if one was raised.

        Returns:
            bool: Always False.
        """
        return False

    def execute(self, statement, parameters=None):
        """Answers one statement from the engine's rows.

        Args:
            statement (sqlalchemy.sql.elements.TextClause): The statement.
            parameters (dict | None): The bound parameters.

        Returns:
            StandInResult: The rows for the statement.
        """
        return self.engine.answer(str(statement), parameters)


class StandInEngine:
    """A stand-in for a SQLAlchemy engine that answers statements from rows held in memory.

    Each statement's rows are narrowed the way its WHERE clause would narrow them: to the broker it names, and to the tokens or instrument ids it lists.

    Attributes:
        answers (list): Pairs of (text fragment, rows); a statement containing the fragment gets the rows.
        calls (list): Pairs of (text fragment, parameters) for every statement answered.
    """

    def __init__(self):
        """Builds an engine with no answers.

        Returns:
            None: This method returns nothing.
        """
        self.answers = []
        self.calls = []

    def add_answer(self, fragment, rows):
        """Adds the rows a statement containing a fragment should return.

        Args:
            fragment (str): Text that appears in the statement and nowhere in the others.
            rows (list): The rows to return.

        Returns:
            None: This method returns nothing.
        """
        self.answers.append((fragment, rows))

    def connect(self):
        """Opens a connection.

        Returns:
            StandInConnection: A new connection.
        """
        return StandInConnection(self)

    def answer(self, statement_text, parameters):
        """Finds the rows for one statement and records the call.

        Args:
            statement_text (str): The statement's SQL.
            parameters (dict | None): The bound parameters.

        Returns:
            StandInResult: The rows for the statement.

        Raises:
            ValueError: If no fragment matches the statement.
        """
        for fragment, rows in self.answers:
            if fragment in statement_text:
                self.calls.append((fragment, parameters))
                kept = []
                for row in rows:
                    if self.keeps(row, parameters):
                        kept.append(row)
                return StandInResult(kept)
        raise ValueError(f'No stand-in answer for statement: {statement_text}')

    def keeps(self, row, parameters):
        """Says whether a row passes a statement's broker, token and instrument id filters.

        Args:
            row (types.SimpleNamespace): The row.
            parameters (dict | None): The bound parameters.

        Returns:
            bool: True when the row passes every filter the parameters carry.
        """
        if parameters is None:
            return True
        if 'broker' in parameters and row.broker != parameters['broker']:
            return False
        if 'tokens' in parameters and row.broker_token not in parameters['tokens']:
            return False
        if 'instrument_identifiers' in parameters and str(row.instrument_id) not in parameters['instrument_identifiers']:
            return False
        return True


class FillingEachTierExample:
    """Resolves two instrument ids cold, warm, and from a second lookup sharing Redis.

    Attributes:
        mapping_date (datetime.date): The mapping date asked about.
        redis_client (StandInRedis): The Redis stand-in.
        engine (StandInEngine): The Postgres stand-in.
        redis_tier (MappingRedisTier): The Redis tier over the stand-in.
        postgres_tier (MappingPostgresTier): The Postgres tier over the stand-in.
        statistics (dict): The counters the lookups share.
        lookup (InstrumentIdentityLookup): The lookup being shown.
    """

    INFY_ID = '0a6e2d41-9c7f-4b58-8d13-6f2e4a9b7c05'
    NIFTY_OPTION_ID = '5f0c7a8e-3b1d-4c62-9e4a-2d8b7f1c0a31'

    def __init__(self):
        """Builds both stand-ins, both tiers and the lookup.

        Returns:
            None: This method returns nothing.
        """
        self.mapping_date = datetime.date(2026, 9, 30)
        self.redis_client = StandInRedis()
        self.engine = StandInEngine()
        self.engine.add_answer('SELECT instrument_id, exchange', [
            types.SimpleNamespace(
                instrument_id=uuid.UUID(self.INFY_ID),
                exchange='nse',
                segment='nse_equities',
                shape='security',
                symbol='INFY',
                underlying_symbol=None,
                expiry_date=None,
                strike_price=None,
                option_type=None,
            ),
            types.SimpleNamespace(
                instrument_id=uuid.UUID(self.NIFTY_OPTION_ID),
                exchange='nse',
                segment='nse_equity_index_options',
                shape='option',
                symbol=None,
                underlying_symbol='NIFTY',
                expiry_date=datetime.date(2026, 10, 6),
                strike_price=decimal.Decimal('25000.00'),
                option_type='CE',
            ),
        ])
        self.redis_tier = MappingRedisTier(MappingRedisConnection(self.redis_client))
        self.postgres_tier = MappingPostgresTier(self.engine)
        self.statistics = {
            'process_hits': 0,
            'redis_hits': 0,
            'postgres_hits': 0,
            'misses': 0,
        }
        self.lookup = InstrumentIdentityLookup(self.redis_tier, self.postgres_tier, self.statistics)

    def describe(self, identity):
        """Describes one identity in a line.

        Args:
            identity (dict): The identity.

        Returns:
            str: The description.
        """
        name = identity['symbol'] or identity['underlying_symbol']
        return f'{identity["segment"]} {name} {identity["expiry_date"]!r} {identity["strike_price"]!r} {identity["option_type"]}'

    def run(self):
        """Resolves the ids three ways, then calls each tier method directly.

        Returns:
            None: This method returns nothing.
        """
        keys = [
            self.INFY_ID,
            self.NIFTY_OPTION_ID,
        ]
        first = self.lookup.resolve(self.mapping_date, keys)
        for key in keys:
            print(f'First resolve: {self.describe(first[key])}')
        print(f'  counters: {self.statistics}')
        print(f'  identities now in Redis: {len(self.redis_client.hgetall(self.redis_tier.identity_key(self.mapping_date)))}')
        self.lookup.resolve(self.mapping_date, keys)
        print(f'Second resolve counters: {self.statistics}')
        second_process = InstrumentIdentityLookup(self.redis_tier, self.postgres_tier, self.statistics)
        from_redis = second_process.resolve(self.mapping_date, keys)
        print(f'Second process: {self.describe(from_redis[self.NIFTY_OPTION_ID])}')
        print(f'  counters: {self.statistics}')
        print(f'Postgres queries so far: {len(self.engine.calls)}')
        option_only = [
            self.NIFTY_OPTION_ID,
        ]
        print(f'read_from_redis finds: {list(self.lookup.read_from_redis(self.mapping_date, option_only))}')
        from_postgres = self.lookup.read_from_postgres(self.mapping_date, option_only)
        print(f'read_from_postgres finds: {list(from_postgres)}')
        print(f'write_to_redis: {self.lookup.write_to_redis(self.mapping_date, from_postgres)}')
        print(f'Postgres queries so far: {len(self.engine.calls)}')


if __name__ == '__main__':
    FillingEachTierExample().run()

It prints:

First resolve: nse_equities INFY None None None
First resolve: nse_equity_index_options NIFTY datetime.date(2026, 10, 6) Decimal('25000.00') CE
  counters: {'process_hits': 0, 'redis_hits': 0, 'postgres_hits': 2, 'misses': 0}
  identities now in Redis: 2
Second resolve counters: {'process_hits': 2, 'redis_hits': 0, 'postgres_hits': 2, 'misses': 0}
Second process: nse_equity_index_options NIFTY datetime.date(2026, 10, 6) Decimal('25000.00') CE
  counters: {'process_hits': 2, 'redis_hits': 2, 'postgres_hits': 2, 'misses': 0}
Postgres queries so far: 1
read_from_redis finds: ['5f0c7a8e-3b1d-4c62-9e4a-2d8b7f1c0a31']
read_from_postgres finds: ['5f0c7a8e-3b1d-4c62-9e4a-2d8b7f1c0a31']
write_to_redis: True
Postgres queries so far: 2
examples/stock_brokers/instruments/mapping/utilities/cache/InstrumentIdentityLookup/example_2_unknown_ids_are_not_remembered.py
"""Shows that an unknown instrument id is asked about again every time, and that a Redis outage only costs a query.

An `InstrumentIdentityLookup` does not remember a miss. An instrument id that `unified.instruments` does not carry is a caller's mistake rather than a fact about today's mapping, and the id may appear with tomorrow's master, so its `empty_entry` stays the base class's None and every look-up of it goes to Postgres.

This program builds the lookup with a Redis tier whose client cannot be reached, because `redis_configuration` is pointed at port 9 on this machine where nothing listens, and a stand-in Postgres engine that holds one real instrument and records the parameters of every query. It asks for that instrument and a mistyped id twice.

Notice three things. The known instrument is answered by Postgres the first time and by process memory the second, even with Redis down, because the process tier needs no Redis. The mistyped id is absent from both answers, counted as a miss both times, and appears in both queries' parameters. And the lookup never raised: an unreachable Redis reads as empty and a failed write answers False.

Run it from the project root:

    python examples/stock_brokers/instruments/mapping/utilities/cache/InstrumentIdentityLookup/example_2_unknown_ids_are_not_remembered.py
"""

import datetime
import types
import uuid

from stock_brokers.instruments.mapping.utilities.cache import (
    InstrumentIdentityLookup,
    MappingPostgresTier,
    MappingRedisTier,
)
from utilities.configurations import redis_configuration


class StandInResult:
    """A stand-in for a SQLAlchemy result holding rows already fetched.

    Attributes:
        rows (list): The rows, in order.
    """

    def __init__(self, rows):
        """Holds the rows.

        Args:
            rows (list): The rows, in order.

        Returns:
            None: This method returns nothing.
        """
        self.rows = rows

    def __iter__(self):
        """Iterates over the rows.

        Returns:
            iterator: An iterator over the rows.
        """
        return iter(self.rows)

    def all(self):
        """Returns every row.

        Returns:
            list: The rows.
        """
        return list(self.rows)

    def scalar(self):
        """Returns the first row's single value.

        Returns:
            object: The first row, which for a scalar answer is the value itself, or None when there is no row.
        """
        if not self.rows:
            return None
        return self.rows[0]


class StandInConnection:
    """A stand-in for a SQLAlchemy connection that asks its engine for each statement's rows.

    Attributes:
        engine (StandInEngine): The engine holding the answers.
    """

    def __init__(self, engine):
        """Builds the connection.

        Args:
            engine (StandInEngine): The engine holding the answers.

        Returns:
            None: This method returns nothing.
        """
        self.engine = engine

    def __enter__(self):
        """Enters a `with` block.

        Returns:
            StandInConnection: This connection.
        """
        return self

    def __exit__(self, exception_type, exception, traceback):
        """Leaves a `with` block without suppressing any exception.

        Args:
            exception_type (type | None): The exception's class, if one was raised.
            exception (BaseException | None): The exception, if one was raised.
            traceback (types.TracebackType | None): The traceback, if one was raised.

        Returns:
            bool: Always False.
        """
        return False

    def execute(self, statement, parameters=None):
        """Answers one statement from the engine's rows.

        Args:
            statement (sqlalchemy.sql.elements.TextClause): The statement.
            parameters (dict | None): The bound parameters.

        Returns:
            StandInResult: The rows for the statement.
        """
        return self.engine.answer(str(statement), parameters)


class StandInEngine:
    """A stand-in for a SQLAlchemy engine that answers statements from rows held in memory.

    Each statement's rows are narrowed the way its WHERE clause would narrow them: to the broker it names, and to the tokens or instrument ids it lists.

    Attributes:
        answers (list): Pairs of (text fragment, rows); a statement containing the fragment gets the rows.
        calls (list): Pairs of (text fragment, parameters) for every statement answered.
    """

    def __init__(self):
        """Builds an engine with no answers.

        Returns:
            None: This method returns nothing.
        """
        self.answers = []
        self.calls = []

    def add_answer(self, fragment, rows):
        """Adds the rows a statement containing a fragment should return.

        Args:
            fragment (str): Text that appears in the statement and nowhere in the others.
            rows (list): The rows to return.

        Returns:
            None: This method returns nothing.
        """
        self.answers.append((fragment, rows))

    def connect(self):
        """Opens a connection.

        Returns:
            StandInConnection: A new connection.
        """
        return StandInConnection(self)

    def answer(self, statement_text, parameters):
        """Finds the rows for one statement and records the call.

        Args:
            statement_text (str): The statement's SQL.
            parameters (dict | None): The bound parameters.

        Returns:
            StandInResult: The rows for the statement.

        Raises:
            ValueError: If no fragment matches the statement.
        """
        for fragment, rows in self.answers:
            if fragment in statement_text:
                self.calls.append((fragment, parameters))
                kept = []
                for row in rows:
                    if self.keeps(row, parameters):
                        kept.append(row)
                return StandInResult(kept)
        raise ValueError(f'No stand-in answer for statement: {statement_text}')

    def keeps(self, row, parameters):
        """Says whether a row passes a statement's broker, token and instrument id filters.

        Args:
            row (types.SimpleNamespace): The row.
            parameters (dict | None): The bound parameters.

        Returns:
            bool: True when the row passes every filter the parameters carry.
        """
        if parameters is None:
            return True
        if 'broker' in parameters and row.broker != parameters['broker']:
            return False
        if 'tokens' in parameters and row.broker_token not in parameters['tokens']:
            return False
        if 'instrument_identifiers' in parameters and str(row.instrument_id) not in parameters['instrument_identifiers']:
            return False
        return True


class UnknownIdsExample:
    """Asks for a known and an unknown instrument id twice, with Redis unreachable.

    Attributes:
        mapping_date (datetime.date): The mapping date asked about.
        engine (StandInEngine): The Postgres stand-in.
        statistics (dict): The counters the lookup counts into.
        lookup (InstrumentIdentityLookup): The lookup being shown.
    """

    RELIANCE_ID = '9e8d7c6b-5a49-4382-a1b0-c9d8e7f6a5b4'
    MISTYPED_ID = '9e8d7c6b-5a49-4382-a1b0-c9d8e7f6a5b5'

    def __init__(self):
        """Points Redis at a closed port and builds the tiers and the lookup.

        Returns:
            None: This method returns nothing.
        """
        redis_configuration['host'] = '127.0.0.1'
        redis_configuration['port'] = 9
        self.mapping_date = datetime.date(2026, 9, 30)
        self.engine = StandInEngine()
        self.engine.add_answer('SELECT instrument_id, exchange', [
            types.SimpleNamespace(
                instrument_id=uuid.UUID(self.RELIANCE_ID),
                exchange='bse',
                segment='bse_equities',
                shape='security',
                symbol='RELIANCE',
                underlying_symbol=None,
                expiry_date=None,
                strike_price=None,
                option_type=None,
            ),
        ])
        self.statistics = {
            'process_hits': 0,
            'redis_hits': 0,
            'postgres_hits': 0,
            'misses': 0,
        }
        self.lookup = InstrumentIdentityLookup(MappingRedisTier(), MappingPostgresTier(self.engine), self.statistics)

    def run(self):
        """Resolves both ids twice and prints the answers, counters and queries.

        Returns:
            None: This method returns nothing.
        """
        keys = [
            self.RELIANCE_ID,
            self.MISTYPED_ID,
        ]
        for attempt in range(1, 3):
            answer = self.lookup.resolve(self.mapping_date, keys)
            symbols = []
            for identity in answer.values():
                symbols.append(f'{identity["exchange"]}:{identity["symbol"]}')
            print(f'Attempt {attempt} answered: {symbols}')
            print(f'  counters: {self.statistics}')
        print(f'Mistyped id remembered: {self.lookup.known(self.MISTYPED_ID)}')
        print(f'empty_entry: {self.lookup.empty_entry()}')
        for fragment, parameters in self.engine.calls:
            print(f'Postgres was asked for: {parameters["instrument_identifiers"]}')
        print(f'read_from_redis with Redis down: {self.lookup.read_from_redis(self.mapping_date, keys)}')
        entries = self.lookup.read_from_postgres(self.mapping_date, keys)
        print(f'write_to_redis with Redis down: {self.lookup.write_to_redis(self.mapping_date, entries)}')


if __name__ == '__main__':
    UnknownIdsExample().run()

It prints:

Attempt 1 answered: ['bse:RELIANCE']
  counters: {'process_hits': 0, 'redis_hits': 0, 'postgres_hits': 1, 'misses': 1}
Attempt 2 answered: ['bse:RELIANCE']
  counters: {'process_hits': 1, 'redis_hits': 0, 'postgres_hits': 1, 'misses': 2}
Mistyped id remembered: False
empty_entry: None
Postgres was asked for: ['9e8d7c6b-5a49-4382-a1b0-c9d8e7f6a5b4', '9e8d7c6b-5a49-4382-a1b0-c9d8e7f6a5b5']
Postgres was asked for: ['9e8d7c6b-5a49-4382-a1b0-c9d8e7f6a5b5']
read_from_redis with Redis down: {}
write_to_redis with Redis down: False

Build the lookup around the two lower tiers and the shared counters.

Parameters:

Name Type Description Default
redis_tier MappingRedisTier

The Redis hashes to read and write.

 required
postgres_tier MappingPostgresTier

The statements to fall through to.

 required
statistics dict

The counters to count into.

 required

Returns:

Type Description
None

This function returns nothing.

Example programs

examples/stock_brokers/instruments/mapping/utilities/cache/ThreeTierLookup/example_1_a_lot_size_lookup.py
"""Builds a small lookup on top of `ThreeTierLookup` and follows one key from Postgres into Redis and into process memory.

`ThreeTierLookup` holds the walk every mapping cache question shares: look in this process's own dictionary, then in Redis, then in Postgres, and copy each answer into the tiers above the one that had it. A subclass supplies only three methods, `read_from_redis`, `read_from_postgres` and `write_to_redis`. The three lookups in `cache.py` are such subclasses; this program writes a fourth, which answers an instrument id with its lot size, so the base class's own behaviour can be seen without the real tiers' detail.

The "Redis" and "Postgres" tiers here are two small stand-in classes holding dictionaries, because the base class never looks inside its tiers and only the subclass talks to them. Notice how the shared `statistics` counters move: the first `resolve` is answered by Postgres, the second by this process, and a second lookup, standing in for another process, is answered by Redis because the first one wrote its answer there. `remember`, `known`, `entry` and `reset` work on the process tier alone and never touch the other two.

Run it from the project root:

    python examples/stock_brokers/instruments/mapping/utilities/cache/ThreeTierLookup/example_1_a_lot_size_lookup.py
"""

import datetime

from stock_brokers.instruments.mapping.utilities.cache import (
    ThreeTierLookup,
)


class DictionaryTier:
    """A stand-in for a Redis or Postgres tier that holds lot sizes in a dictionary.

    Attributes:
        lot_sizes (dict): Each instrument id's lot size.
        reads (int): How many times the tier was read.
    """

    def __init__(self, lot_sizes):
        """Builds the tier.

        Args:
            lot_sizes (dict): Each instrument id's lot size.

        Returns:
            None: This method returns nothing.
        """
        self.lot_sizes = lot_sizes
        self.reads = 0

    def read(self, keys):
        """Reads the lot sizes of the keys this tier holds.

        Args:
            keys (list): The instrument ids to read.

        Returns:
            dict: Each held instrument id to an entry with its lot size.
        """
        self.reads += 1
        found = {}
        for key in keys:
            if key in self.lot_sizes:
                found[key] = {
                    'lot_size': self.lot_sizes[key],
                }
        return found

    def write(self, entries):
        """Stores entries' lot sizes.

        Args:
            entries (dict): Each instrument id to an entry with its lot size.

        Returns:
            bool: Always True.
        """
        for key, entry in entries.items():
            self.lot_sizes[key] = entry['lot_size']
        return True


class LotSizeLookup(ThreeTierLookup):
    """Instrument id to that instrument's lot size, through the three tiers."""

    def read_from_redis(self, mapping_date, keys):
        """Reads lot sizes out of the Redis stand-in.

        Args:
            mapping_date (datetime.date): The mapping date to read.
            keys (list): The instrument ids to read.

        Returns:
            dict: Each held instrument id to its entry.
        """
        return self.redis_tier.read(keys)

    def read_from_postgres(self, mapping_date, keys):
        """Reads lot sizes out of the Postgres stand-in.

        Args:
            mapping_date (datetime.date): The mapping date to read.
            keys (list): The instrument ids to read.

        Returns:
            dict: Each held instrument id to its entry.
        """
        return self.postgres_tier.read(keys)

    def write_to_redis(self, mapping_date, entries):
        """Writes lot sizes into the Redis stand-in.

        Args:
            mapping_date (datetime.date): The mapping date the entries belong to.
            entries (dict): Each instrument id to its entry.

        Returns:
            bool: True when the write went through.
        """
        return self.redis_tier.write(entries)


class LotSizeLookupExample:
    """Resolves lot sizes through a first and a second lookup sharing one Redis stand-in.

    Attributes:
        mapping_date (datetime.date): The mapping date asked about.
        redis_tier (DictionaryTier): The Redis stand-in, empty at first.
        postgres_tier (DictionaryTier): The Postgres stand-in, holding every lot size.
        statistics (dict): The counters the lookups share.
        lookup (LotSizeLookup): The first process's lookup.
    """

    def __init__(self):
        """Builds the tiers, the counters and the first lookup.

        Returns:
            None: This method returns nothing.
        """
        self.mapping_date = datetime.date(2026, 9, 30)
        self.redis_tier = DictionaryTier({})
        self.postgres_tier = DictionaryTier({
            'nifty-future': '75',
            'banknifty-future': '35',
            'crudeoil-future': '100',
        })
        self.statistics = {
            'process_hits': 0,
            'redis_hits': 0,
            'postgres_hits': 0,
            'misses': 0,
        }
        self.lookup = LotSizeLookup(self.redis_tier, self.postgres_tier, self.statistics)

    def run(self):
        """Resolves the same keys cold, warm and from a second lookup, then uses the process tier directly.

        Returns:
            None: This method returns nothing.
        """
        keys = [
            'nifty-future',
            'banknifty-future',
        ]
        print(f'First resolve: {self.lookup.resolve(self.mapping_date, keys)}')
        print(f'  counters: {self.statistics}')
        print(f'Second resolve: {self.lookup.resolve(self.mapping_date, keys)}')
        print(f'  counters: {self.statistics}')
        print(f'Redis now holds: {self.redis_tier.lot_sizes}')
        second_process = LotSizeLookup(self.redis_tier, self.postgres_tier, self.statistics)
        print(f'Second process resolves: {second_process.resolve(self.mapping_date, keys)}')
        print(f'  counters: {self.statistics}')
        print(f'Postgres was read {self.postgres_tier.reads} time')
        print(f'Known before remember: {self.lookup.known("crudeoil-future")}')
        crude_entry = {
            'lot_size': '100',
        }
        self.lookup.remember('crudeoil-future', crude_entry)
        print(f'Known after remember: {self.lookup.known("crudeoil-future")}')
        print(f'Entry held for it: {self.lookup.entry("crudeoil-future")}')
        print(f'Entry for a key never seen: {self.lookup.entry("goldm-future")}')
        print(f'What a miss is remembered as: {self.lookup.empty_entry()}')
        crude_only = [
            'crudeoil-future',
        ]
        print(f'Reading Redis directly: {self.lookup.read_from_redis(self.mapping_date, crude_only)}')
        print(f'Reading Postgres directly: {self.lookup.read_from_postgres(self.mapping_date, crude_only)}')
        crude_entries = {
            'crudeoil-future': crude_entry,
        }
        print(f'Writing Redis directly: {self.lookup.write_to_redis(self.mapping_date, crude_entries)}')
        self.lookup.reset()
        print(f'Known after reset: {self.lookup.known("nifty-future")}')


if __name__ == '__main__':
    LotSizeLookupExample().run()

It prints:

First resolve: {'nifty-future': {'lot_size': '75'}, 'banknifty-future': {'lot_size': '35'}}
  counters: {'process_hits': 0, 'redis_hits': 0, 'postgres_hits': 2, 'misses': 0}
Second resolve: {'nifty-future': {'lot_size': '75'}, 'banknifty-future': {'lot_size': '35'}}
  counters: {'process_hits': 2, 'redis_hits': 0, 'postgres_hits': 2, 'misses': 0}
Redis now holds: {'nifty-future': '75', 'banknifty-future': '35'}
Second process resolves: {'nifty-future': {'lot_size': '75'}, 'banknifty-future': {'lot_size': '35'}}
  counters: {'process_hits': 2, 'redis_hits': 2, 'postgres_hits': 2, 'misses': 0}
Postgres was read 1 time
Known before remember: False
Known after remember: True
Entry held for it: {'lot_size': '100'}
Entry for a key never seen: None
What a miss is remembered as: None
Reading Redis directly: {}
Reading Postgres directly: {'crudeoil-future': {'lot_size': '100'}}
Writing Redis directly: True
Known after reset: False
examples/stock_brokers/instruments/mapping/utilities/cache/ThreeTierLookup/example_2_remembering_misses.py
"""Shows what the base class does on its own, and how `empty_entry` decides whether a miss is asked about again.

`ThreeTierLookup` is meant to be subclassed. Used directly, its three tier methods raise `NotImplementedError` naming the class, while an empty batch of keys is answered without reaching any tier at all.

The more interesting choice a subclass makes is `empty_entry`. When Postgres has no answer for a key, the walk asks `empty_entry()` what to remember. The base answer, None, remembers nothing, so the next look-up of that key goes all the way to Postgres again; the cache's identity lookup works this way, because an unknown instrument id is a caller's mistake that tomorrow's master may fix. A subclass that returns an empty value instead makes the miss a fact for the rest of the day; the cache's token lookup works this way, because a token a broker does not carry is asked about on every tick.

This program builds two small subclasses that differ only in `empty_entry` and asks each about the same unknown key three times, against a stand-in Postgres that counts its reads. Notice that the first asks Postgres three times and the second once.

Run it from the project root:

    python examples/stock_brokers/instruments/mapping/utilities/cache/ThreeTierLookup/example_2_remembering_misses.py
"""

import datetime

from stock_brokers.instruments.mapping.utilities.cache import (
    ThreeTierLookup,
)


class CountingEmptyTier:
    """A stand-in tier that holds nothing and counts how often it is read.

    Attributes:
        reads (int): How many times the tier was read.
    """

    def __init__(self):
        """Builds the tier with no reads counted.

        Returns:
            None: This method returns nothing.
        """
        self.reads = 0

    def read(self, keys):
        """Counts one read and finds nothing.

        Args:
            keys (list): The keys asked about.

        Returns:
            dict: Always empty.
        """
        self.reads += 1
        return {}


class ForgettingLookup(ThreeTierLookup):
    """A lookup that keeps the base class's `empty_entry`, so it never remembers a miss."""

    def read_from_redis(self, mapping_date, keys):
        """Reads the Redis stand-in.

        Args:
            mapping_date (datetime.date): The mapping date to read.
            keys (list): The keys to read.

        Returns:
            dict: What the stand-in held.
        """
        return self.redis_tier.read(keys)

    def read_from_postgres(self, mapping_date, keys):
        """Reads the Postgres stand-in.

        Args:
            mapping_date (datetime.date): The mapping date to read.
            keys (list): The keys to read.

        Returns:
            dict: What the stand-in held.
        """
        return self.postgres_tier.read(keys)

    def write_to_redis(self, mapping_date, entries):
        """Accepts entries without storing them.

        Args:
            mapping_date (datetime.date): The mapping date the entries belong to.
            entries (dict): The entries to write.

        Returns:
            bool: Always True.
        """
        return True


class RememberingLookup(ForgettingLookup):
    """A lookup that remembers a miss as an empty list for the rest of the day."""

    def empty_entry(self):
        """What to remember for a key nothing maps.

        Returns:
            list: A new empty list.
        """
        return []


class RememberingMissesExample:
    """Calls the base class directly, then compares a forgetting and a remembering lookup.

    Attributes:
        mapping_date (datetime.date): The mapping date asked about.
        statistics (dict): The counters the lookups count into.
    """

    def __init__(self):
        """Builds the date and the counters.

        Returns:
            None: This method returns nothing.
        """
        self.mapping_date = datetime.date(2026, 9, 30)
        self.statistics = {
            'process_hits': 0,
            'redis_hits': 0,
            'postgres_hits': 0,
            'misses': 0,
        }

    def show_base_class(self):
        """Calls each method of the base class itself and prints what happens.

        Returns:
            None: This method returns nothing.
        """
        base = ThreeTierLookup(None, None, self.statistics)
        print(f'Base resolve of no keys: {base.resolve(self.mapping_date, [])}')
        print(f'Base empty_entry: {base.empty_entry()}')
        keys = [
            '999999',
        ]
        try:
            base.resolve(self.mapping_date, keys)
        except NotImplementedError as error:
            print(f'Base resolve of a key: NotImplementedError: {error}')
        try:
            base.read_from_redis(self.mapping_date, keys)
        except NotImplementedError as error:
            print(f'Base read_from_redis: NotImplementedError: {error}')
        try:
            base.read_from_postgres(self.mapping_date, keys)
        except NotImplementedError as error:
            print(f'Base read_from_postgres: NotImplementedError: {error}')
        try:
            base.write_to_redis(self.mapping_date, {})
        except NotImplementedError as error:
            print(f'Base write_to_redis: NotImplementedError: {error}')

    def ask_three_times(self, lookup, postgres_tier):
        """Asks one lookup about an unknown key three times and prints the result.

        Args:
            lookup (ThreeTierLookup): The lookup to ask.
            postgres_tier (CountingEmptyTier): The Postgres stand-in behind it.

        Returns:
            None: This method returns nothing.
        """
        keys = [
            '999999',
        ]
        answers = []
        for attempt in range(3):
            answers.append(lookup.resolve(self.mapping_date, keys))
        name = type(lookup).__name__
        print(f'{name}: answers {answers}, Postgres read {postgres_tier.reads} times, remembered: {lookup.known("999999")}')

    def run(self):
        """Shows the base class, then the two lookups side by side.

        Returns:
            None: This method returns nothing.
        """
        self.show_base_class()
        forgetting_postgres = CountingEmptyTier()
        forgetting = ForgettingLookup(CountingEmptyTier(), forgetting_postgres, self.statistics)
        self.ask_three_times(forgetting, forgetting_postgres)
        remembering_postgres = CountingEmptyTier()
        remembering = RememberingLookup(CountingEmptyTier(), remembering_postgres, self.statistics)
        self.ask_three_times(remembering, remembering_postgres)
        print(f'Counters: {self.statistics}')


if __name__ == '__main__':
    RememberingMissesExample().run()

It prints:

Base resolve of no keys: {}
Base empty_entry: None
Base resolve of a key: NotImplementedError: ThreeTierLookup must implement read_from_redis()
Base read_from_redis: NotImplementedError: ThreeTierLookup must implement read_from_redis()
Base read_from_postgres: NotImplementedError: ThreeTierLookup must implement read_from_postgres()
Base write_to_redis: NotImplementedError: ThreeTierLookup must implement write_to_redis()
ForgettingLookup: answers [{}, {}, {}], Postgres read 3 times, remembered: False
RememberingLookup: answers [{'999999': []}, {'999999': []}, {'999999': []}], Postgres read 1 times, remembered: True
Counters: {'process_hits': 2, 'redis_hits': 0, 'postgres_hits': 0, 'misses': 4}

Methods:

Name Description
read_from_redis

Read identities out of the identity hash.
read_from_postgres

Read identities out of unified.instruments.
write_to_redis

Write identities back into the identity hash.
resolve

Answer for a batch of keys, trying each tier in turn and filling in the ones above.
remember

Put one entry into this process's memory without going to any other tier.
known

Whether this process already holds an entry for one key.
entry

The entry this process holds for one key, without reaching any other tier.
reset

Drop everything this lookup holds in process memory.
empty_entry

What to remember for a key nothing maps, or None to remember nothing.
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1637
1638
1639
1640
1641
1642
1643
1644
1645
1646
1647
1648
1649
1650
1651
1652
1653
def __init__(self, redis_tier, postgres_tier, statistics):
    """
    Build the lookup around the two lower tiers and the shared counters.

    Args:
        redis_tier (MappingRedisTier): The Redis hashes to read and write.
        postgres_tier (MappingPostgresTier): The statements to fall through to.
        statistics (dict): The counters to count into.

    Returns:
        None: This function returns nothing.
    """
    self.redis_tier = redis_tier
    self.postgres_tier = postgres_tier
    self.statistics = statistics

    self._entries = {}

read_from_redis 

read_from_redis(mapping_date, keys)

Read identities out of the identity hash.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required
keys list[str]

The instrument ids to read.

 required

Returns:

Type Description
dict

Mapping of instrument id to its identity dict.

Example usage

From example_1_filling_each_tier_in_turn.py
print(f'read_from_redis finds: {list(self.lookup.read_from_redis(self.mapping_date, option_only))}')
From example_2_unknown_ids_are_not_remembered.py
print(f'read_from_redis with Redis down: {self.lookup.read_from_redis(self.mapping_date, keys)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1830
1831
1832
1833
1834
1835
1836
1837
1838
1839
1840
1841
def read_from_redis(self, mapping_date, keys):
    """
    Read identities out of the identity hash.

    Args:
        mapping_date (datetime.date): The mapping date to read.
        keys (list[str]): The instrument ids to read.

    Returns:
        dict: Mapping of instrument id to its identity dict.
    """
    return self.redis_tier.read_identities(mapping_date, keys)

read_from_postgres 

read_from_postgres(mapping_date, keys)

Read identities out of unified.instruments.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to stamp on each identity.

 required
keys list[str]

The instrument ids to read.

 required

Returns:

Type Description
dict

Mapping of instrument id to its identity dict.

Example usage

From example_1_filling_each_tier_in_turn.py
from_postgres = self.lookup.read_from_postgres(self.mapping_date, option_only)
From example_2_unknown_ids_are_not_remembered.py
entries = self.lookup.read_from_postgres(self.mapping_date, keys)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1843
1844
1845
1846
1847
1848
1849
1850
1851
1852
1853
1854
def read_from_postgres(self, mapping_date, keys):
    """
    Read identities out of unified.instruments.

    Args:
        mapping_date (datetime.date): The mapping date to stamp on each identity.
        keys (list[str]): The instrument ids to read.

    Returns:
        dict: Mapping of instrument id to its identity dict.
    """
    return self.postgres_tier.read_identities(mapping_date, keys)

write_to_redis 

write_to_redis(mapping_date, entries)

Write identities back into the identity hash.

Parameters:

Name Type Description Default
mapping_date date

The mapping date the identities belong to.

 required
entries dict

Mapping of instrument id to its identity dict.

 required

Returns:

Type Description
bool

True when the write went through, False when Redis was not usable.

Example usage

From example_1_filling_each_tier_in_turn.py
print(f'write_to_redis: {self.lookup.write_to_redis(self.mapping_date, from_postgres)}')
From example_2_unknown_ids_are_not_remembered.py
print(f'write_to_redis with Redis down: {self.lookup.write_to_redis(self.mapping_date, entries)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1856
1857
1858
1859
1860
1861
1862
1863
1864
1865
1866
1867
def write_to_redis(self, mapping_date, entries):
    """
    Write identities back into the identity hash.

    Args:
        mapping_date (datetime.date): The mapping date the identities belong to.
        entries (dict): Mapping of instrument id to its identity dict.

    Returns:
        bool: True when the write went through, False when Redis was not usable.
    """
    return self.redis_tier.write_identities(mapping_date, entries)

resolve 

resolve(mapping_date, keys)

Answer for a batch of keys, trying each tier in turn and filling in the ones above.

A key that nothing maps is remembered as an empty entry when the subclass says so, which stops a token nothing carries being asked about on every call. A subclass whose empty_entry is None does not remember misses at all.

Parameters:

Name Type Description Default
mapping_date date

The mapping date being answered for.

 required
keys list[str]

The keys to answer for.

 required

Returns:

Type Description
dict

Mapping of key to its entry, for every key that had one. Keys nothing maps are absent, unless the subclass remembers empty entries, in which case they carry one.

Example usage

From example_1_a_lot_size_lookup.py
print(f'First resolve: {self.lookup.resolve(self.mapping_date, keys)}')
From example_1_a_lot_size_lookup.py
print(f'Second resolve: {self.lookup.resolve(self.mapping_date, keys)}')
From example_1_a_lot_size_lookup.py
print(f'Second process resolves: {second_process.resolve(self.mapping_date, keys)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1655
1656
1657
1658
1659
1660
1661
1662
1663
1664
1665
1666
1667
1668
1669
1670
1671
1672
1673
1674
1675
1676
1677
1678
1679
1680
1681
1682
1683
1684
1685
1686
1687
1688
1689
1690
1691
1692
1693
1694
1695
1696
1697
1698
1699
1700
1701
1702
1703
1704
1705
1706
1707
1708
1709
def resolve(self, mapping_date, keys):
    """
    Answer for a batch of keys, trying each tier in turn and filling in the ones above.

    A key that nothing maps is remembered as an empty entry when the subclass says so, which stops a token nothing carries being asked about on every call. A subclass whose empty_entry is None does not remember misses at all.

    Args:
        mapping_date (datetime.date): The mapping date being answered for.
        keys (list[str]): The keys to answer for.

    Returns:
        dict: Mapping of key to its entry, for every key that had one. Keys nothing maps are absent, unless the subclass remembers empty entries, in which case they carry one.
    """
    if not keys:
        return {}

    resolved = {}
    outstanding = []
    for key in keys:
        entry = self._entries.get(key)
        if entry is None:
            outstanding.append(key)
            continue
        self.statistics["process_hits"] += 1
        resolved[key] = entry

    if outstanding:
        from_redis = self.read_from_redis(mapping_date, outstanding)
        still_outstanding = []
        for key in outstanding:
            if key not in from_redis:
                still_outstanding.append(key)
                continue
            self.statistics["redis_hits"] += 1
            self._entries[key] = from_redis[key]
            resolved[key] = from_redis[key]
        outstanding = still_outstanding

    if outstanding:
        from_postgres = self.read_from_postgres(mapping_date, outstanding)
        for key in outstanding:
            entry = from_postgres.get(key)
            if entry is None:
                self.statistics["misses"] += 1
                empty = self.empty_entry()
                if empty is not None:
                    self._entries[key] = empty
                    resolved[key] = empty
                continue
            self.statistics["postgres_hits"] += 1
            self._entries[key] = entry
            resolved[key] = entry
        self.write_to_redis(mapping_date, from_postgres)

    return resolved

remember 

remember(key, entry)

Put one entry into this process's memory without going to any other tier.

This exists so that a lookup filling itself can also fill a sibling, which is how a token's candidate identities land in the identity lookup rather than being read again from Redis.

Parameters:

Name Type Description Default
key str

The key to remember the entry under.

 required
entry dict

The entry to remember.

 required

Returns:

Type Description
None

This function returns nothing.

Example usage

From example_1_a_lot_size_lookup.py
self.lookup.remember('crudeoil-future', crude_entry)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1711
1712
1713
1714
1715
1716
1717
1718
1719
1720
1721
1722
1723
1724
def remember(self, key, entry):
    """
    Put one entry into this process's memory without going to any other tier.

    This exists so that a lookup filling itself can also fill a sibling, which is how a token's candidate identities land in the identity lookup rather than being read again from Redis.

    Args:
        key (str): The key to remember the entry under.
        entry (dict): The entry to remember.

    Returns:
        None: This function returns nothing.
    """
    self._entries[key] = entry

known 

known(key)

Whether this process already holds an entry for one key.

Parameters:

Name Type Description Default
key str

The key to test.

 required

Returns:

Type Description
bool

True when the entry is in this process's memory.

Example usage

From example_1_a_lot_size_lookup.py
print(f'Known before remember: {self.lookup.known("crudeoil-future")}')
From example_1_a_lot_size_lookup.py
print(f'Known after remember: {self.lookup.known("crudeoil-future")}')
From example_1_a_lot_size_lookup.py
print(f'Known after reset: {self.lookup.known("nifty-future")}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1726
1727
1728
1729
1730
1731
1732
1733
1734
1735
1736
def known(self, key):
    """
    Whether this process already holds an entry for one key.

    Args:
        key (str): The key to test.

    Returns:
        bool: True when the entry is in this process's memory.
    """
    return key in self._entries

entry 

entry(key)

The entry this process holds for one key, without reaching any other tier.

The counterpart of remember, for a sibling lookup assembling an answer out of entries this one already has.

Parameters:

Name Type Description Default
key str

The key to read.

 required

Returns:

Type Description
dict | None

The entry, or None when this process does not hold one.

Example usage

From example_1_a_lot_size_lookup.py
print(f'Entry held for it: {self.lookup.entry("crudeoil-future")}')
From example_1_a_lot_size_lookup.py
print(f'Entry for a key never seen: {self.lookup.entry("goldm-future")}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1738
1739
1740
1741
1742
1743
1744
1745
1746
1747
1748
1749
1750
def entry(self, key):
    """
    The entry this process holds for one key, without reaching any other tier.

    The counterpart of remember, for a sibling lookup assembling an answer out of entries this one already has.

    Args:
        key (str): The key to read.

    Returns:
        dict | None: The entry, or None when this process does not hold one.
    """
    return self._entries.get(key)

reset 

reset()

Drop everything this lookup holds in process memory.

Returns:

Type Description
None

This function returns nothing.

Example usage

From example_1_a_lot_size_lookup.py
self.lookup.reset()
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1752
1753
1754
1755
1756
1757
1758
1759
def reset(self):
    """
    Drop everything this lookup holds in process memory.

    Returns:
        None: This function returns nothing.
    """
    self._entries.clear()

empty_entry 

empty_entry()

What to remember for a key nothing maps, or None to remember nothing.

A fresh object is returned on every call rather than a shared one, so that an entry a caller narrows in place cannot reach the next key that also missed.

Returns:

Type Description
None

The base class does not remember misses.

Example usage

From example_1_a_lot_size_lookup.py
print(f'What a miss is remembered as: {self.lookup.empty_entry()}')
From example_2_remembering_misses.py
print(f'Base empty_entry: {base.empty_entry()}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1761
1762
1763
1764
1765
1766
1767
1768
1769
1770
def empty_entry(self):
    """
    What to remember for a key nothing maps, or None to remember nothing.

    A fresh object is returned on every call rather than a shared one, so that an entry a caller narrows in place cannot reach the next key that also missed.

    Returns:
        None: The base class does not remember misses.
    """
    return None

TokenCandidateLookup 

TokenCandidateLookup(redis_tier, postgres_tier, statistics, broker, identity_lookup)

Bases: ThreeTierLookup

One broker's token to every identity that token carries, for one broker.

A lookup is built per broker rather than one for all ten, so its memory is keyed by the token alone and the shared walk needs no notion of a compound key.

A broker token can carry more than one instrument on the same date, which is why the candidates are stored in full rather than resolved once and stored as a winner. Once an optional segment filter exists the same token has two right answers depending on the filter, and caching the resolved winner would let a filtered and an unfiltered look-up contaminate each other. Storing them all is what lets both share one entry, and the filter is applied on the way out.

Attributes:

Name Type Description
broker str

The broker whose tokens this lookup answers for.
identity_lookup InstrumentIdentityLookup

The identity store this lookup shares, so an instrument seen from either direction costs one read.

Build the lookup for one broker.

Parameters:

Name Type Description Default
redis_tier MappingRedisTier

The Redis hashes to read and write.

 required
postgres_tier MappingPostgresTier

The statements to fall through to.

 required
statistics dict

The counters to count into.

 required
broker str

The broker whose tokens this lookup answers for.

 required
identity_lookup InstrumentIdentityLookup

The identity store to share.

 required

Returns:

Type Description
None

This function returns nothing.

Example programs

examples/stock_brokers/instruments/mapping/utilities/cache/TokenCandidateLookup/example_1_one_token_two_instruments.py
"""Resolves Dhan security ids to every instrument each one carries, and shares what it learns with the identity lookup.

A `TokenCandidateLookup` answers the question a market feed asks on every tick: which unified instruments does this broker token stand for? It is built for one broker, and it keeps the whole list of candidates for a token rather than a single winner, because one token can carry more than one instrument on the same day. Dhan numbers each exchange's securities separately, so in this program's made-up rows the security id 1594 is INFY on the NSE and another company on the BSE. A caller that filters by segment later picks the right one; storing only a winner would give a filtered and an unfiltered caller the same wrong answer.

The lookup also shares an `InstrumentIdentityLookup`. Every identity it reads on the way past is remembered there, so an instrument first met through a token costs nothing when it is later asked about by id.

Two stand-ins replace the data stores: a Redis client that keeps hashes in dictionaries, and a Postgres engine that holds the rows of the token statement, keeps only the broker and tokens a statement asks for, and records every call. Notice that token 1594 comes back with two candidates in tie-break order, which is by symbol and then by instrument id, that the Redis token hash stores their instrument ids joined by a comma, and that the identity lookup knows all three instruments afterwards without having been asked.

Run it from the project root:

    python examples/stock_brokers/instruments/mapping/utilities/cache/TokenCandidateLookup/example_1_one_token_two_instruments.py
"""

import datetime
import types
import uuid

from stock_brokers.instruments.mapping.utilities.cache import (
    InstrumentIdentityLookup,
    MappingPostgresTier,
    MappingRedisConnection,
    MappingRedisTier,
    TokenCandidateLookup,
)


class StandInPipeline:
    """A stand-in for a Redis pipeline that queues commands and runs them on execute.

    Attributes:
        redis_client (StandInRedis): The stand-in client the commands run against.
        queued (list): The queued commands, as (method, arguments) pairs.
    """

    def __init__(self, redis_client):
        """Builds an empty pipeline.

        Args:
            redis_client (StandInRedis): The stand-in client the commands run against.

        Returns:
            None: This method returns nothing.
        """
        self.redis_client = redis_client
        self.queued = []

    def set(self, key, value):
        """Queues a string write.

        Args:
            key (str): The key to write.
            value (str): The value to store.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.set, (key, value)))

    def hset(self, key, mapping):
        """Queues a hash write.

        Args:
            key (str): The hash to write into.
            mapping (dict): The fields to write.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.hset, (key, mapping)))

    def hmget(self, key, fields):
        """Queues a read of several hash fields.

        Args:
            key (str): The hash to read.
            fields (list): The fields to read.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.hmget, (key, fields)))

    def expire(self, key, seconds):
        """Queues an expiry.

        Args:
            key (str): The key to expire.
            seconds (int): How long the key may live.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.expire, (key, seconds)))

    def execute(self):
        """Runs every queued command in order.

        Returns:
            list: Each command's reply, in the order queued.
        """
        replies = []
        for method, arguments in self.queued:
            replies.append(method(*arguments))
        self.queued = []
        return replies


class StandInRedis:
    """A stand-in for a Redis client that keeps strings and hashes in dictionaries.

    Attributes:
        values (dict): Each key's value: a string, or a dict for a hash.
        expiries (dict): Each key's expiry in seconds, as last set.
    """

    def __init__(self):
        """Builds an empty stand-in.

        Returns:
            None: This method returns nothing.
        """
        self.values = {}
        self.expiries = {}

    def pipeline(self, transaction=True):
        """Starts a pipeline.

        Args:
            transaction (bool): Ignored; the stand-in always runs commands in order.

        Returns:
            StandInPipeline: A new pipeline.
        """
        return StandInPipeline(self)

    def get(self, key):
        """Reads a string.

        Args:
            key (str): The key to read.

        Returns:
            str | None: The value, or None when the key is absent.
        """
        return self.values.get(key)

    def set(self, key, value):
        """Writes a string.

        Args:
            key (str): The key to write.
            value (str): The value to store.

        Returns:
            bool: Always True.
        """
        self.values[key] = value
        return True

    def exists(self, key):
        """Counts whether a key exists.

        Args:
            key (str): The key to test.

        Returns:
            int: 1 when the key exists, otherwise 0.
        """
        if key in self.values:
            return 1
        return 0

    def expire(self, key, seconds):
        """Records a key's expiry.

        Args:
            key (str): The key to expire.
            seconds (int): How long the key may live.

        Returns:
            bool: Always True.
        """
        self.expiries[key] = seconds
        return True

    def hset(self, key, mapping):
        """Writes fields into a hash.

        Args:
            key (str): The hash to write into.
            mapping (dict): The fields to write.

        Returns:
            int: The number of fields written.
        """
        stored = self.values.setdefault(key, {})
        stored.update(mapping)
        return len(mapping)

    def hmget(self, key, fields):
        """Reads several fields of a hash.

        Args:
            key (str): The hash to read.
            fields (list): The fields to read.

        Returns:
            list: Each field's value, or None where the field is absent.
        """
        stored = self.values.get(key, {})
        replies = []
        for field in fields:
            replies.append(stored.get(field))
        return replies

    def hgetall(self, key):
        """Reads a whole hash.

        Args:
            key (str): The hash to read.

        Returns:
            dict: The hash's fields, empty when the key is absent.
        """
        return dict(self.values.get(key, {}))


class StandInResult:
    """A stand-in for a SQLAlchemy result holding rows already fetched.

    Attributes:
        rows (list): The rows, in order.
    """

    def __init__(self, rows):
        """Holds the rows.

        Args:
            rows (list): The rows, in order.

        Returns:
            None: This method returns nothing.
        """
        self.rows = rows

    def __iter__(self):
        """Iterates over the rows.

        Returns:
            iterator: An iterator over the rows.
        """
        return iter(self.rows)

    def all(self):
        """Returns every row.

        Returns:
            list: The rows.
        """
        return list(self.rows)

    def scalar(self):
        """Returns the first row's single value.

        Returns:
            object: The first row, which for a scalar answer is the value itself, or None when there is no row.
        """
        if not self.rows:
            return None
        return self.rows[0]


class StandInConnection:
    """A stand-in for a SQLAlchemy connection that asks its engine for each statement's rows.

    Attributes:
        engine (StandInEngine): The engine holding the answers.
    """

    def __init__(self, engine):
        """Builds the connection.

        Args:
            engine (StandInEngine): The engine holding the answers.

        Returns:
            None: This method returns nothing.
        """
        self.engine = engine

    def __enter__(self):
        """Enters a `with` block.

        Returns:
            StandInConnection: This connection.
        """
        return self

    def __exit__(self, exception_type, exception, traceback):
        """Leaves a `with` block without suppressing any exception.

        Args:
            exception_type (type | None): The exception's class, if one was raised.
            exception (BaseException | None): The exception, if one was raised.
            traceback (types.TracebackType | None): The traceback, if one was raised.

        Returns:
            bool: Always False.
        """
        return False

    def execute(self, statement, parameters=None):
        """Answers one statement from the engine's rows.

        Args:
            statement (sqlalchemy.sql.elements.TextClause): The statement.
            parameters (dict | None): The bound parameters.

        Returns:
            StandInResult: The rows for the statement.
        """
        return self.engine.answer(str(statement), parameters)


class StandInEngine:
    """A stand-in for a SQLAlchemy engine that answers statements from rows held in memory.

    Each statement's rows are narrowed the way its WHERE clause would narrow them: to the broker it names, and to the tokens or instrument ids it lists.

    Attributes:
        answers (list): Pairs of (text fragment, rows); a statement containing the fragment gets the rows.
        calls (list): Pairs of (text fragment, parameters) for every statement answered.
    """

    def __init__(self):
        """Builds an engine with no answers.

        Returns:
            None: This method returns nothing.
        """
        self.answers = []
        self.calls = []

    def add_answer(self, fragment, rows):
        """Adds the rows a statement containing a fragment should return.

        Args:
            fragment (str): Text that appears in the statement and nowhere in the others.
            rows (list): The rows to return.

        Returns:
            None: This method returns nothing.
        """
        self.answers.append((fragment, rows))

    def connect(self):
        """Opens a connection.

        Returns:
            StandInConnection: A new connection.
        """
        return StandInConnection(self)

    def answer(self, statement_text, parameters):
        """Finds the rows for one statement and records the call.

        Args:
            statement_text (str): The statement's SQL.
            parameters (dict | None): The bound parameters.

        Returns:
            StandInResult: The rows for the statement.

        Raises:
            ValueError: If no fragment matches the statement.
        """
        for fragment, rows in self.answers:
            if fragment in statement_text:
                self.calls.append((fragment, parameters))
                kept = []
                for row in rows:
                    if self.keeps(row, parameters):
                        kept.append(row)
                return StandInResult(kept)
        raise ValueError(f'No stand-in answer for statement: {statement_text}')

    def keeps(self, row, parameters):
        """Says whether a row passes a statement's broker, token and instrument id filters.

        Args:
            row (types.SimpleNamespace): The row.
            parameters (dict | None): The bound parameters.

        Returns:
            bool: True when the row passes every filter the parameters carry.
        """
        if parameters is None:
            return True
        if 'broker' in parameters and row.broker != parameters['broker']:
            return False
        if 'tokens' in parameters and row.broker_token not in parameters['tokens']:
            return False
        if 'instrument_identifiers' in parameters and str(row.instrument_id) not in parameters['instrument_identifiers']:
            return False
        return True


class OneTokenTwoInstrumentsExample:
    """Resolves two Dhan tokens, one of which carries two instruments.

    Attributes:
        mapping_date (datetime.date): The mapping date asked about.
        redis_client (StandInRedis): The Redis stand-in.
        engine (StandInEngine): The Postgres stand-in.
        redis_tier (MappingRedisTier): The Redis tier over the stand-in.
        statistics (dict): The counters the lookups share.
        identity_lookup (InstrumentIdentityLookup): The identity store the token lookup fills.
        lookup (TokenCandidateLookup): The lookup being shown, for Dhan.
    """

    INFY_NSE_ID = '0a6e2d41-9c7f-4b58-8d13-6f2e4a9b7c05'
    BSE_1594_ID = '6c4a2e08-1f3d-4b5a-9c7e-0d2f4a6c8e13'
    TCS_NSE_ID = '8f2b4d6a-0c1e-4a3b-8d5f-7a9c1e3b5d24'

    def __init__(self):
        """Builds the stand-ins, the tiers and both lookups.

        Returns:
            None: This method returns nothing.
        """
        self.mapping_date = datetime.date(2026, 9, 30)
        self.redis_client = StandInRedis()
        self.engine = StandInEngine()
        self.engine.add_answer('b.broker_token = ANY', [
            self.candidate_row('11536', self.TCS_NSE_ID, 'nse', 'nse_equities', 'TCS'),
            self.candidate_row('1594', self.BSE_1594_ID, 'bse', 'bse_equities', 'GANESHHOU'),
            self.candidate_row('1594', self.INFY_NSE_ID, 'nse', 'nse_equities', 'INFY'),
        ])
        self.redis_tier = MappingRedisTier(MappingRedisConnection(self.redis_client))
        postgres_tier = MappingPostgresTier(self.engine)
        self.statistics = {
            'process_hits': 0,
            'redis_hits': 0,
            'postgres_hits': 0,
            'misses': 0,
        }
        self.identity_lookup = InstrumentIdentityLookup(self.redis_tier, postgres_tier, self.statistics)
        self.lookup = TokenCandidateLookup(self.redis_tier, postgres_tier, self.statistics, 'dhan', self.identity_lookup)

    def candidate_row(self, broker_token, instrument_identifier, exchange, segment, symbol):
        """Builds one row of the token candidate join for an equity.

        Args:
            broker_token (str): The broker's token.
            instrument_identifier (str): The instrument id.
            exchange (str): The exchange.
            segment (str): The exchange-prefixed segment.
            symbol (str): The symbol.

        Returns:
            types.SimpleNamespace: The row.
        """
        return types.SimpleNamespace(
            broker='dhan',
            broker_token=broker_token,
            mapping_date=self.mapping_date,
            instrument_id=uuid.UUID(instrument_identifier),
            exchange=exchange,
            segment=segment,
            shape='security',
            symbol=symbol,
            underlying_symbol=None,
            expiry_date=None,
            strike_price=None,
            option_type=None,
        )

    def run(self):
        """Resolves the tokens, then shows what Redis and the identity lookup now hold.

        Returns:
            None: This method returns nothing.
        """
        tokens = [
            '1594',
            '11536',
        ]
        print(f'Lookup for broker: {self.lookup.broker}')
        answer = self.lookup.resolve(self.mapping_date, tokens)
        for token in tokens:
            for identity in answer[token]:
                print(f'Token {token}: {identity["segment"]} {identity["symbol"]}')
        print(f'  counters: {self.statistics}')
        stored_tokens = self.redis_client.hgetall(self.redis_tier.tokens_key(self.mapping_date, 'dhan'))
        print(f'Redis token hash: {stored_tokens}')
        instrument_identifiers = [
            self.INFY_NSE_ID,
            self.BSE_1594_ID,
            self.TCS_NSE_ID,
        ]
        for instrument_identifier in instrument_identifiers:
            print(f'Identity lookup knows {instrument_identifier}: {self.lookup.identity_lookup.known(instrument_identifier)}')
        print(f'empty_entry: {self.lookup.empty_entry()}')
        tcs_only = [
            '11536',
        ]
        from_postgres = self.lookup.read_from_postgres(self.mapping_date, tcs_only)
        print(f'read_from_postgres 11536: {from_postgres["11536"][0]["symbol"]}')
        print(f'write_to_redis: {self.lookup.write_to_redis(self.mapping_date, from_postgres)}')
        from_redis = self.lookup.read_from_redis(self.mapping_date, tcs_only)
        print(f'read_from_redis 11536: {from_redis["11536"][0]["symbol"]}')
        print(f'Postgres queries: {len(self.engine.calls)}')


if __name__ == '__main__':
    OneTokenTwoInstrumentsExample().run()

It prints:

Lookup for broker: dhan
Token 1594: bse_equities GANESHHOU
Token 1594: nse_equities INFY
Token 11536: nse_equities TCS
  counters: {'process_hits': 0, 'redis_hits': 0, 'postgres_hits': 2, 'misses': 0}
Redis token hash: {'11536': '8f2b4d6a-0c1e-4a3b-8d5f-7a9c1e3b5d24', '1594': '6c4a2e08-1f3d-4b5a-9c7e-0d2f4a6c8e13,0a6e2d41-9c7f-4b58-8d13-6f2e4a9b7c05'}
Identity lookup knows 0a6e2d41-9c7f-4b58-8d13-6f2e4a9b7c05: True
Identity lookup knows 6c4a2e08-1f3d-4b5a-9c7e-0d2f4a6c8e13: True
Identity lookup knows 8f2b4d6a-0c1e-4a3b-8d5f-7a9c1e3b5d24: True
empty_entry: []
read_from_postgres 11536: TCS
write_to_redis: True
read_from_redis 11536: TCS
Postgres queries: 2
examples/stock_brokers/instruments/mapping/utilities/cache/TokenCandidateLookup/example_2_reading_what_another_process_left.py
"""Reads tokens that an earlier process left in Redis, and remembers a token no instrument carries.

This is what a second process sees after the first has done the work. The program first writes two Dhan tokens and their identities into Redis through `MappingRedisTier.write_token_candidates`, as the daily warm or an earlier process would, and then deletes one of the identities from the Redis identity hash to imitate an entry that expired early. A fresh `TokenCandidateLookup` then resolves both tokens and a third token that Dhan does not carry at all.

Its `read_from_redis` reads the token hash, fetches only the identities the shared identity lookup does not already hold, and returns a token only when every one of its candidates was found, because a token missing a candidate could silently give a segment-filtered caller the wrong instrument. So the token with a missing identity falls through to Postgres and is refilled, while the complete one is a Redis hit. The unknown token is remembered as an empty list, its `empty_entry`, so asking about it again costs nothing.

Two stand-ins replace the data stores: a Redis client that keeps hashes in dictionaries, and a Postgres engine that holds the rows of the token statement, keeps only the broker and tokens a statement asks for, and records every call. Notice which tokens reached Postgres, and that the second round is answered entirely from this process.

Run it from the project root:

    python examples/stock_brokers/instruments/mapping/utilities/cache/TokenCandidateLookup/example_2_reading_what_another_process_left.py
"""

import datetime
import types
import uuid

from stock_brokers.instruments.mapping.utilities.cache import (
    InstrumentIdentityLookup,
    MappingPostgresTier,
    MappingRedisConnection,
    MappingRedisTier,
    TokenCandidateLookup,
)


class StandInPipeline:
    """A stand-in for a Redis pipeline that queues commands and runs them on execute.

    Attributes:
        redis_client (StandInRedis): The stand-in client the commands run against.
        queued (list): The queued commands, as (method, arguments) pairs.
    """

    def __init__(self, redis_client):
        """Builds an empty pipeline.

        Args:
            redis_client (StandInRedis): The stand-in client the commands run against.

        Returns:
            None: This method returns nothing.
        """
        self.redis_client = redis_client
        self.queued = []

    def set(self, key, value):
        """Queues a string write.

        Args:
            key (str): The key to write.
            value (str): The value to store.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.set, (key, value)))

    def hset(self, key, mapping):
        """Queues a hash write.

        Args:
            key (str): The hash to write into.
            mapping (dict): The fields to write.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.hset, (key, mapping)))

    def hmget(self, key, fields):
        """Queues a read of several hash fields.

        Args:
            key (str): The hash to read.
            fields (list): The fields to read.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.hmget, (key, fields)))

    def expire(self, key, seconds):
        """Queues an expiry.

        Args:
            key (str): The key to expire.
            seconds (int): How long the key may live.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.expire, (key, seconds)))

    def execute(self):
        """Runs every queued command in order.

        Returns:
            list: Each command's reply, in the order queued.
        """
        replies = []
        for method, arguments in self.queued:
            replies.append(method(*arguments))
        self.queued = []
        return replies


class StandInRedis:
    """A stand-in for a Redis client that keeps strings and hashes in dictionaries.

    Attributes:
        values (dict): Each key's value: a string, or a dict for a hash.
        expiries (dict): Each key's expiry in seconds, as last set.
    """

    def __init__(self):
        """Builds an empty stand-in.

        Returns:
            None: This method returns nothing.
        """
        self.values = {}
        self.expiries = {}

    def pipeline(self, transaction=True):
        """Starts a pipeline.

        Args:
            transaction (bool): Ignored; the stand-in always runs commands in order.

        Returns:
            StandInPipeline: A new pipeline.
        """
        return StandInPipeline(self)

    def get(self, key):
        """Reads a string.

        Args:
            key (str): The key to read.

        Returns:
            str | None: The value, or None when the key is absent.
        """
        return self.values.get(key)

    def set(self, key, value):
        """Writes a string.

        Args:
            key (str): The key to write.
            value (str): The value to store.

        Returns:
            bool: Always True.
        """
        self.values[key] = value
        return True

    def exists(self, key):
        """Counts whether a key exists.

        Args:
            key (str): The key to test.

        Returns:
            int: 1 when the key exists, otherwise 0.
        """
        if key in self.values:
            return 1
        return 0

    def expire(self, key, seconds):
        """Records a key's expiry.

        Args:
            key (str): The key to expire.
            seconds (int): How long the key may live.

        Returns:
            bool: Always True.
        """
        self.expiries[key] = seconds
        return True

    def hset(self, key, mapping):
        """Writes fields into a hash.

        Args:
            key (str): The hash to write into.
            mapping (dict): The fields to write.

        Returns:
            int: The number of fields written.
        """
        stored = self.values.setdefault(key, {})
        stored.update(mapping)
        return len(mapping)

    def hmget(self, key, fields):
        """Reads several fields of a hash.

        Args:
            key (str): The hash to read.
            fields (list): The fields to read.

        Returns:
            list: Each field's value, or None where the field is absent.
        """
        stored = self.values.get(key, {})
        replies = []
        for field in fields:
            replies.append(stored.get(field))
        return replies

    def hgetall(self, key):
        """Reads a whole hash.

        Args:
            key (str): The hash to read.

        Returns:
            dict: The hash's fields, empty when the key is absent.
        """
        return dict(self.values.get(key, {}))


class StandInResult:
    """A stand-in for a SQLAlchemy result holding rows already fetched.

    Attributes:
        rows (list): The rows, in order.
    """

    def __init__(self, rows):
        """Holds the rows.

        Args:
            rows (list): The rows, in order.

        Returns:
            None: This method returns nothing.
        """
        self.rows = rows

    def __iter__(self):
        """Iterates over the rows.

        Returns:
            iterator: An iterator over the rows.
        """
        return iter(self.rows)

    def all(self):
        """Returns every row.

        Returns:
            list: The rows.
        """
        return list(self.rows)

    def scalar(self):
        """Returns the first row's single value.

        Returns:
            object: The first row, which for a scalar answer is the value itself, or None when there is no row.
        """
        if not self.rows:
            return None
        return self.rows[0]


class StandInConnection:
    """A stand-in for a SQLAlchemy connection that asks its engine for each statement's rows.

    Attributes:
        engine (StandInEngine): The engine holding the answers.
    """

    def __init__(self, engine):
        """Builds the connection.

        Args:
            engine (StandInEngine): The engine holding the answers.

        Returns:
            None: This method returns nothing.
        """
        self.engine = engine

    def __enter__(self):
        """Enters a `with` block.

        Returns:
            StandInConnection: This connection.
        """
        return self

    def __exit__(self, exception_type, exception, traceback):
        """Leaves a `with` block without suppressing any exception.

        Args:
            exception_type (type | None): The exception's class, if one was raised.
            exception (BaseException | None): The exception, if one was raised.
            traceback (types.TracebackType | None): The traceback, if one was raised.

        Returns:
            bool: Always False.
        """
        return False

    def execute(self, statement, parameters=None):
        """Answers one statement from the engine's rows.

        Args:
            statement (sqlalchemy.sql.elements.TextClause): The statement.
            parameters (dict | None): The bound parameters.

        Returns:
            StandInResult: The rows for the statement.
        """
        return self.engine.answer(str(statement), parameters)


class StandInEngine:
    """A stand-in for a SQLAlchemy engine that answers statements from rows held in memory.

    Each statement's rows are narrowed the way its WHERE clause would narrow them: to the broker it names, and to the tokens or instrument ids it lists.

    Attributes:
        answers (list): Pairs of (text fragment, rows); a statement containing the fragment gets the rows.
        calls (list): Pairs of (text fragment, parameters) for every statement answered.
    """

    def __init__(self):
        """Builds an engine with no answers.

        Returns:
            None: This method returns nothing.
        """
        self.answers = []
        self.calls = []

    def add_answer(self, fragment, rows):
        """Adds the rows a statement containing a fragment should return.

        Args:
            fragment (str): Text that appears in the statement and nowhere in the others.
            rows (list): The rows to return.

        Returns:
            None: This method returns nothing.
        """
        self.answers.append((fragment, rows))

    def connect(self):
        """Opens a connection.

        Returns:
            StandInConnection: A new connection.
        """
        return StandInConnection(self)

    def answer(self, statement_text, parameters):
        """Finds the rows for one statement and records the call.

        Args:
            statement_text (str): The statement's SQL.
            parameters (dict | None): The bound parameters.

        Returns:
            StandInResult: The rows for the statement.

        Raises:
            ValueError: If no fragment matches the statement.
        """
        for fragment, rows in self.answers:
            if fragment in statement_text:
                self.calls.append((fragment, parameters))
                kept = []
                for row in rows:
                    if self.keeps(row, parameters):
                        kept.append(row)
                return StandInResult(kept)
        raise ValueError(f'No stand-in answer for statement: {statement_text}')

    def keeps(self, row, parameters):
        """Says whether a row passes a statement's broker, token and instrument id filters.

        Args:
            row (types.SimpleNamespace): The row.
            parameters (dict | None): The bound parameters.

        Returns:
            bool: True when the row passes every filter the parameters carry.
        """
        if parameters is None:
            return True
        if 'broker' in parameters and row.broker != parameters['broker']:
            return False
        if 'tokens' in parameters and row.broker_token not in parameters['tokens']:
            return False
        if 'instrument_identifiers' in parameters and str(row.instrument_id) not in parameters['instrument_identifiers']:
            return False
        return True


class ReadingWhatAnotherProcessLeftExample:
    """Fills Redis as an earlier process would, damages one entry, and resolves through a fresh lookup.

    Attributes:
        mapping_date (datetime.date): The mapping date asked about.
        redis_client (StandInRedis): The Redis stand-in.
        engine (StandInEngine): The Postgres stand-in.
        redis_tier (MappingRedisTier): The Redis tier over the stand-in.
        postgres_tier (MappingPostgresTier): The Postgres tier over the stand-in.
        statistics (dict): The counters the lookups share.
        lookup (TokenCandidateLookup): The fresh lookup being shown, for Dhan.
    """

    INFY_NSE_ID = '0a6e2d41-9c7f-4b58-8d13-6f2e4a9b7c05'
    TCS_NSE_ID = '8f2b4d6a-0c1e-4a3b-8d5f-7a9c1e3b5d24'

    def __init__(self):
        """Builds the stand-ins, the tiers and a fresh lookup.

        Returns:
            None: This method returns nothing.
        """
        self.mapping_date = datetime.date(2026, 9, 30)
        self.redis_client = StandInRedis()
        self.engine = StandInEngine()
        self.engine.add_answer('b.broker_token = ANY', [
            self.candidate_row('11536', self.TCS_NSE_ID, 'nse', 'nse_equities', 'TCS'),
            self.candidate_row('1594', self.INFY_NSE_ID, 'nse', 'nse_equities', 'INFY'),
        ])
        self.redis_tier = MappingRedisTier(MappingRedisConnection(self.redis_client))
        self.postgres_tier = MappingPostgresTier(self.engine)
        self.statistics = {
            'process_hits': 0,
            'redis_hits': 0,
            'postgres_hits': 0,
            'misses': 0,
        }
        identity_lookup = InstrumentIdentityLookup(self.redis_tier, self.postgres_tier, self.statistics)
        self.lookup = TokenCandidateLookup(self.redis_tier, self.postgres_tier, self.statistics, 'dhan', identity_lookup)

    def fill_redis_as_another_process(self):
        """Writes both tokens into Redis, then deletes TCS's identity from the identity hash.

        Returns:
            None: This method returns nothing.
        """
        both_tokens = [
            '1594',
            '11536',
        ]
        candidates_by_token = self.postgres_tier.read_token_candidates(self.mapping_date, 'dhan', both_tokens)
        self.redis_tier.write_token_candidates(self.mapping_date, 'dhan', candidates_by_token)
        identity_hash = self.redis_client.values[self.redis_tier.identity_key(self.mapping_date)]
        del identity_hash[self.TCS_NSE_ID]
        self.engine.calls = []
        print(f'Redis holds tokens: {sorted(self.redis_client.values[self.redis_tier.tokens_key(self.mapping_date, "dhan")])}')
        print(f'Redis holds identities for: {sorted(identity_hash)}')

    def candidate_row(self, broker_token, instrument_identifier, exchange, segment, symbol):
        """Builds one row of the token candidate join for an equity.

        Args:
            broker_token (str): The broker's token.
            instrument_identifier (str): The instrument id.
            exchange (str): The exchange.
            segment (str): The exchange-prefixed segment.
            symbol (str): The symbol.

        Returns:
            types.SimpleNamespace: The row.
        """
        return types.SimpleNamespace(
            broker='dhan',
            broker_token=broker_token,
            mapping_date=self.mapping_date,
            instrument_id=uuid.UUID(instrument_identifier),
            exchange=exchange,
            segment=segment,
            shape='security',
            symbol=symbol,
            underlying_symbol=None,
            expiry_date=None,
            strike_price=None,
            option_type=None,
        )

    def run(self):
        """Resolves three tokens twice through the fresh lookup and prints what each round cost.

        Returns:
            None: This method returns nothing.
        """
        self.fill_redis_as_another_process()
        tokens = [
            '1594',
            '11536',
            '999999',
        ]
        only_complete = self.lookup.read_from_redis(self.mapping_date, tokens)
        print(f'read_from_redis returns only complete tokens: {sorted(only_complete)}')
        answer = self.lookup.resolve(self.mapping_date, tokens)
        for token in tokens:
            symbols = []
            for identity in answer[token]:
                symbols.append(identity['symbol'])
            print(f'Token {token}: {symbols}')
        print(f'  counters: {self.statistics}')
        for fragment, parameters in self.engine.calls:
            print(f'  Postgres was asked for tokens: {parameters["tokens"]}')
        self.lookup.resolve(self.mapping_date, tokens)
        print(f'Second round counters: {self.statistics}')
        print(f'Unknown token remembered: {self.lookup.known("999999")} as {self.lookup.entry("999999")}')
        print(f'A fresh empty_entry is a new list: {self.lookup.empty_entry() is not self.lookup.entry("999999")}')
        refilled = self.lookup.read_from_postgres(self.mapping_date, tokens)
        print(f'read_from_postgres finds tokens: {sorted(refilled)}')
        print(f'write_to_redis: {self.lookup.write_to_redis(self.mapping_date, refilled)}')


if __name__ == '__main__':
    ReadingWhatAnotherProcessLeftExample().run()

It prints:

Redis holds tokens: ['11536', '1594']
Redis holds identities for: ['0a6e2d41-9c7f-4b58-8d13-6f2e4a9b7c05']
read_from_redis returns only complete tokens: ['1594']
Token 1594: ['INFY']
Token 11536: ['TCS']
Token 999999: []
  counters: {'process_hits': 0, 'redis_hits': 1, 'postgres_hits': 1, 'misses': 1}
  Postgres was asked for tokens: ['11536', '999999']
Second round counters: {'process_hits': 3, 'redis_hits': 1, 'postgres_hits': 1, 'misses': 1}
Unknown token remembered: True as []
A fresh empty_entry is a new list: True
read_from_postgres finds tokens: ['11536', '1594']
write_to_redis: True

Methods:

Name Description
empty_entry

What to remember for a token nothing maps.
read_from_redis

Read tokens out of the token hash, resolving each to its full list of candidate identities.
read_from_postgres

Read tokens out of Postgres, with every candidate identity each token carries.
write_to_redis

Write tokens and the identities they carry back into Redis.
resolve

Answer for a batch of keys, trying each tier in turn and filling in the ones above.
remember

Put one entry into this process's memory without going to any other tier.
known

Whether this process already holds an entry for one key.
entry

The entry this process holds for one key, without reaching any other tier.
reset

Drop everything this lookup holds in process memory.
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1883
1884
1885
1886
1887
1888
1889
1890
1891
1892
1893
1894
1895
1896
1897
1898
1899
def __init__(self, redis_tier, postgres_tier, statistics, broker, identity_lookup):
    """
    Build the lookup for one broker.

    Args:
        redis_tier (MappingRedisTier): The Redis hashes to read and write.
        postgres_tier (MappingPostgresTier): The statements to fall through to.
        statistics (dict): The counters to count into.
        broker (str): The broker whose tokens this lookup answers for.
        identity_lookup (InstrumentIdentityLookup): The identity store to share.

    Returns:
        None: This function returns nothing.
    """
    super().__init__(redis_tier, postgres_tier, statistics)
    self.broker = broker
    self.identity_lookup = identity_lookup

empty_entry 

empty_entry()

What to remember for a token nothing maps.

An empty candidate list is remembered, because a token a broker does not carry is a stable fact for the day and asking Postgres about it on every call would be the cost this cache exists to remove.

Returns:

Type Description
list

A new empty list.

Example usage

From example_1_one_token_two_instruments.py
print(f'empty_entry: {self.lookup.empty_entry()}')
From example_2_reading_what_another_process_left.py
print(f'A fresh empty_entry is a new list: {self.lookup.empty_entry() is not self.lookup.entry("999999")}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1901
1902
1903
1904
1905
1906
1907
1908
1909
1910
def empty_entry(self):
    """
    What to remember for a token nothing maps.

    An empty candidate list is remembered, because a token a broker does not carry is a stable fact for the day and asking Postgres about it on every call would be the cost this cache exists to remove.

    Returns:
        list: A new empty list.
    """
    return []

read_from_redis 

read_from_redis(mapping_date, keys)

Read tokens out of the token hash, resolving each to its full list of candidate identities.

The identities this process already holds are not read again, and only a token whose every candidate resolved is returned, because a partly resolved token would silently lose a candidate the segment filter might have wanted.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required
keys list[str]

The broker tokens to read.

 required

Returns:

Type Description
dict

Mapping of broker token to a list of identity dicts.

Example usage

From example_1_one_token_two_instruments.py
from_redis = self.lookup.read_from_redis(self.mapping_date, tcs_only)
From example_2_reading_what_another_process_left.py
only_complete = self.lookup.read_from_redis(self.mapping_date, tokens)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1912
1913
1914
1915
1916
1917
1918
1919
1920
1921
1922
1923
1924
1925
1926
1927
1928
1929
1930
1931
1932
1933
1934
1935
1936
1937
1938
1939
1940
1941
1942
1943
1944
1945
1946
1947
1948
1949
1950
1951
1952
1953
1954
def read_from_redis(self, mapping_date, keys):
    """
    Read tokens out of the token hash, resolving each to its full list of candidate identities.

    The identities this process already holds are not read again, and only a token whose every candidate resolved is returned, because a partly resolved token would silently lose a candidate the segment filter might have wanted.

    Args:
        mapping_date (datetime.date): The mapping date to read.
        keys (list[str]): The broker tokens to read.

    Returns:
        dict: Mapping of broker token to a list of identity dicts.
    """
    identifiers_by_token = self.redis_tier.read_token_identifiers(mapping_date, self.broker, keys)
    if not identifiers_by_token:
        return {}

    wanted_identifiers = []
    already_wanted = set()
    for identifiers in identifiers_by_token.values():
        for instrument_identifier in identifiers:
            if self.identity_lookup.known(instrument_identifier):
                continue
            if instrument_identifier in already_wanted:
                continue
            already_wanted.add(instrument_identifier)
            wanted_identifiers.append(instrument_identifier)

    if wanted_identifiers:
        from_redis = self.redis_tier.read_identities(mapping_date, wanted_identifiers)
        for instrument_identifier, identity in from_redis.items():
            self.identity_lookup.remember(instrument_identifier, identity)

    candidates_by_token = {}
    for token, identifiers in identifiers_by_token.items():
        candidates = []
        for instrument_identifier in identifiers:
            identity = self.identity_lookup.entry(instrument_identifier)
            if identity is not None:
                candidates.append(identity)
        if len(candidates) == len(identifiers):
            candidates_by_token[token] = candidates
    return candidates_by_token

read_from_postgres 

read_from_postgres(mapping_date, keys)

Read tokens out of Postgres, with every candidate identity each token carries.

The identities are remembered in the shared identity store on the way past, so an instrument reached first through a token costs nothing when it is later asked about by id.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required
keys list[str]

The broker tokens to read.

 required

Returns:

Type Description
dict

Mapping of broker token to a list of identity dicts, in tie-break order.

Example usage

From example_1_one_token_two_instruments.py
from_postgres = self.lookup.read_from_postgres(self.mapping_date, tcs_only)
From example_2_reading_what_another_process_left.py
refilled = self.lookup.read_from_postgres(self.mapping_date, tokens)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1956
1957
1958
1959
1960
1961
1962
1963
1964
1965
1966
1967
1968
1969
1970
1971
1972
1973
def read_from_postgres(self, mapping_date, keys):
    """
    Read tokens out of Postgres, with every candidate identity each token carries.

    The identities are remembered in the shared identity store on the way past, so an instrument reached first through a token costs nothing when it is later asked about by id.

    Args:
        mapping_date (datetime.date): The mapping date to read.
        keys (list[str]): The broker tokens to read.

    Returns:
        dict: Mapping of broker token to a list of identity dicts, in tie-break order.
    """
    candidates_by_token = self.postgres_tier.read_token_candidates(mapping_date, self.broker, keys)
    for candidates in candidates_by_token.values():
        for identity in candidates:
            self.identity_lookup.remember(identity["instrument_id"], identity)
    return candidates_by_token

write_to_redis 

write_to_redis(mapping_date, entries)

Write tokens and the identities they carry back into Redis.

Parameters:

Name Type Description Default
mapping_date date

The mapping date the answers belong to.

 required
entries dict

Mapping of broker token to a list of identity dicts.

 required

Returns:

Type Description
bool

True when the write went through, False when Redis was not usable.

Example usage

From example_1_one_token_two_instruments.py
print(f'write_to_redis: {self.lookup.write_to_redis(self.mapping_date, from_postgres)}')
From example_2_reading_what_another_process_left.py
print(f'write_to_redis: {self.lookup.write_to_redis(self.mapping_date, refilled)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1975
1976
1977
1978
1979
1980
1981
1982
1983
1984
1985
1986
def write_to_redis(self, mapping_date, entries):
    """
    Write tokens and the identities they carry back into Redis.

    Args:
        mapping_date (datetime.date): The mapping date the answers belong to.
        entries (dict): Mapping of broker token to a list of identity dicts.

    Returns:
        bool: True when the write went through, False when Redis was not usable.
    """
    return self.redis_tier.write_token_candidates(mapping_date, self.broker, entries)

resolve 

resolve(mapping_date, keys)

Answer for a batch of keys, trying each tier in turn and filling in the ones above.

A key that nothing maps is remembered as an empty entry when the subclass says so, which stops a token nothing carries being asked about on every call. A subclass whose empty_entry is None does not remember misses at all.

Parameters:

Name Type Description Default
mapping_date date

The mapping date being answered for.

 required
keys list[str]

The keys to answer for.

 required

Returns:

Type Description
dict

Mapping of key to its entry, for every key that had one. Keys nothing maps are absent, unless the subclass remembers empty entries, in which case they carry one.

Example usage

From example_1_a_lot_size_lookup.py
print(f'First resolve: {self.lookup.resolve(self.mapping_date, keys)}')
From example_1_a_lot_size_lookup.py
print(f'Second resolve: {self.lookup.resolve(self.mapping_date, keys)}')
From example_1_a_lot_size_lookup.py
print(f'Second process resolves: {second_process.resolve(self.mapping_date, keys)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1655
1656
1657
1658
1659
1660
1661
1662
1663
1664
1665
1666
1667
1668
1669
1670
1671
1672
1673
1674
1675
1676
1677
1678
1679
1680
1681
1682
1683
1684
1685
1686
1687
1688
1689
1690
1691
1692
1693
1694
1695
1696
1697
1698
1699
1700
1701
1702
1703
1704
1705
1706
1707
1708
1709
def resolve(self, mapping_date, keys):
    """
    Answer for a batch of keys, trying each tier in turn and filling in the ones above.

    A key that nothing maps is remembered as an empty entry when the subclass says so, which stops a token nothing carries being asked about on every call. A subclass whose empty_entry is None does not remember misses at all.

    Args:
        mapping_date (datetime.date): The mapping date being answered for.
        keys (list[str]): The keys to answer for.

    Returns:
        dict: Mapping of key to its entry, for every key that had one. Keys nothing maps are absent, unless the subclass remembers empty entries, in which case they carry one.
    """
    if not keys:
        return {}

    resolved = {}
    outstanding = []
    for key in keys:
        entry = self._entries.get(key)
        if entry is None:
            outstanding.append(key)
            continue
        self.statistics["process_hits"] += 1
        resolved[key] = entry

    if outstanding:
        from_redis = self.read_from_redis(mapping_date, outstanding)
        still_outstanding = []
        for key in outstanding:
            if key not in from_redis:
                still_outstanding.append(key)
                continue
            self.statistics["redis_hits"] += 1
            self._entries[key] = from_redis[key]
            resolved[key] = from_redis[key]
        outstanding = still_outstanding

    if outstanding:
        from_postgres = self.read_from_postgres(mapping_date, outstanding)
        for key in outstanding:
            entry = from_postgres.get(key)
            if entry is None:
                self.statistics["misses"] += 1
                empty = self.empty_entry()
                if empty is not None:
                    self._entries[key] = empty
                    resolved[key] = empty
                continue
            self.statistics["postgres_hits"] += 1
            self._entries[key] = entry
            resolved[key] = entry
        self.write_to_redis(mapping_date, from_postgres)

    return resolved

remember 

remember(key, entry)

Put one entry into this process's memory without going to any other tier.

This exists so that a lookup filling itself can also fill a sibling, which is how a token's candidate identities land in the identity lookup rather than being read again from Redis.

Parameters:

Name Type Description Default
key str

The key to remember the entry under.

 required
entry dict

The entry to remember.

 required

Returns:

Type Description
None

This function returns nothing.

Example usage

From example_1_a_lot_size_lookup.py
self.lookup.remember('crudeoil-future', crude_entry)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1711
1712
1713
1714
1715
1716
1717
1718
1719
1720
1721
1722
1723
1724
def remember(self, key, entry):
    """
    Put one entry into this process's memory without going to any other tier.

    This exists so that a lookup filling itself can also fill a sibling, which is how a token's candidate identities land in the identity lookup rather than being read again from Redis.

    Args:
        key (str): The key to remember the entry under.
        entry (dict): The entry to remember.

    Returns:
        None: This function returns nothing.
    """
    self._entries[key] = entry

known 

known(key)

Whether this process already holds an entry for one key.

Parameters:

Name Type Description Default
key str

The key to test.

 required

Returns:

Type Description
bool

True when the entry is in this process's memory.

Example usage

From example_1_a_lot_size_lookup.py
print(f'Known before remember: {self.lookup.known("crudeoil-future")}')
From example_1_a_lot_size_lookup.py
print(f'Known after remember: {self.lookup.known("crudeoil-future")}')
From example_1_a_lot_size_lookup.py
print(f'Known after reset: {self.lookup.known("nifty-future")}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1726
1727
1728
1729
1730
1731
1732
1733
1734
1735
1736
def known(self, key):
    """
    Whether this process already holds an entry for one key.

    Args:
        key (str): The key to test.

    Returns:
        bool: True when the entry is in this process's memory.
    """
    return key in self._entries

entry 

entry(key)

The entry this process holds for one key, without reaching any other tier.

The counterpart of remember, for a sibling lookup assembling an answer out of entries this one already has.

Parameters:

Name Type Description Default
key str

The key to read.

 required

Returns:

Type Description
dict | None

The entry, or None when this process does not hold one.

Example usage

From example_1_a_lot_size_lookup.py
print(f'Entry held for it: {self.lookup.entry("crudeoil-future")}')
From example_1_a_lot_size_lookup.py
print(f'Entry for a key never seen: {self.lookup.entry("goldm-future")}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1738
1739
1740
1741
1742
1743
1744
1745
1746
1747
1748
1749
1750
def entry(self, key):
    """
    The entry this process holds for one key, without reaching any other tier.

    The counterpart of remember, for a sibling lookup assembling an answer out of entries this one already has.

    Args:
        key (str): The key to read.

    Returns:
        dict | None: The entry, or None when this process does not hold one.
    """
    return self._entries.get(key)

reset 

reset()

Drop everything this lookup holds in process memory.

Returns:

Type Description
None

This function returns nothing.

Example usage

From example_1_a_lot_size_lookup.py
self.lookup.reset()
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1752
1753
1754
1755
1756
1757
1758
1759
def reset(self):
    """
    Drop everything this lookup holds in process memory.

    Returns:
        None: This function returns nothing.
    """
    self._entries.clear()

OrderHandleLookup 

OrderHandleLookup(redis_tier, postgres_tier, statistics)

Bases: ThreeTierLookup

Instrument id to every broker's order handle for that instrument.

A handle is everything an order needs to name an instrument at one broker: the broker's own token, the tradeable symbol where that broker orders by symbol rather than by token, and the lot and tick sizes an order has to be a multiple of. All four come out of one row, so an order placement costs one dictionary look-up once this process has seen the instrument, and one query the first time it has not.

Example programs

examples/stock_brokers/instruments/mapping/utilities/cache/OrderHandleLookup/example_1_handles_for_an_order.py
"""Resolves an instrument to every broker's order handle, the four things an order needs to name it at that broker.

An `OrderHandleLookup` answers the question the order route asks on every order: for this unified instrument, what token, tradeable symbol, lot size and tick size does each broker want? All four come from one row of `unified.broker_mappings` per broker, so the first look-up costs one query, and every later one in the same process is a dictionary read.

Two stand-ins replace the data stores: a Redis client that keeps hashes in dictionaries, and a Postgres engine that holds the rows of the order handle statement, keeps only the instrument ids a statement asks for, and records every call. The real `MappingRedisTier` and `MappingPostgresTier` sit on top of them.

Notice that Zerodha orders a future by its trading symbol while Dhan orders it by its security id alone, so Dhan's `order_symbol` is None; that lot and tick sizes are text, so they survive being stored as JSON exactly; and that a second lookup, standing in for another gunicorn worker, gets the handles from Redis without a query.

Run it from the project root:

    python examples/stock_brokers/instruments/mapping/utilities/cache/OrderHandleLookup/example_1_handles_for_an_order.py
"""

import datetime
import decimal
import types
import uuid

from stock_brokers.instruments.mapping.utilities.cache import (
    MappingPostgresTier,
    MappingRedisConnection,
    MappingRedisTier,
    OrderHandleLookup,
)


class StandInPipeline:
    """A stand-in for a Redis pipeline that queues commands and runs them on execute.

    Attributes:
        redis_client (StandInRedis): The stand-in client the commands run against.
        queued (list): The queued commands, as (method, arguments) pairs.
    """

    def __init__(self, redis_client):
        """Builds an empty pipeline.

        Args:
            redis_client (StandInRedis): The stand-in client the commands run against.

        Returns:
            None: This method returns nothing.
        """
        self.redis_client = redis_client
        self.queued = []

    def set(self, key, value):
        """Queues a string write.

        Args:
            key (str): The key to write.
            value (str): The value to store.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.set, (key, value)))

    def hset(self, key, mapping):
        """Queues a hash write.

        Args:
            key (str): The hash to write into.
            mapping (dict): The fields to write.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.hset, (key, mapping)))

    def hmget(self, key, fields):
        """Queues a read of several hash fields.

        Args:
            key (str): The hash to read.
            fields (list): The fields to read.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.hmget, (key, fields)))

    def expire(self, key, seconds):
        """Queues an expiry.

        Args:
            key (str): The key to expire.
            seconds (int): How long the key may live.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.expire, (key, seconds)))

    def execute(self):
        """Runs every queued command in order.

        Returns:
            list: Each command's reply, in the order queued.
        """
        replies = []
        for method, arguments in self.queued:
            replies.append(method(*arguments))
        self.queued = []
        return replies


class StandInRedis:
    """A stand-in for a Redis client that keeps strings and hashes in dictionaries.

    Attributes:
        values (dict): Each key's value: a string, or a dict for a hash.
        expiries (dict): Each key's expiry in seconds, as last set.
    """

    def __init__(self):
        """Builds an empty stand-in.

        Returns:
            None: This method returns nothing.
        """
        self.values = {}
        self.expiries = {}

    def pipeline(self, transaction=True):
        """Starts a pipeline.

        Args:
            transaction (bool): Ignored; the stand-in always runs commands in order.

        Returns:
            StandInPipeline: A new pipeline.
        """
        return StandInPipeline(self)

    def get(self, key):
        """Reads a string.

        Args:
            key (str): The key to read.

        Returns:
            str | None: The value, or None when the key is absent.
        """
        return self.values.get(key)

    def set(self, key, value):
        """Writes a string.

        Args:
            key (str): The key to write.
            value (str): The value to store.

        Returns:
            bool: Always True.
        """
        self.values[key] = value
        return True

    def exists(self, key):
        """Counts whether a key exists.

        Args:
            key (str): The key to test.

        Returns:
            int: 1 when the key exists, otherwise 0.
        """
        if key in self.values:
            return 1
        return 0

    def expire(self, key, seconds):
        """Records a key's expiry.

        Args:
            key (str): The key to expire.
            seconds (int): How long the key may live.

        Returns:
            bool: Always True.
        """
        self.expiries[key] = seconds
        return True

    def hset(self, key, mapping):
        """Writes fields into a hash.

        Args:
            key (str): The hash to write into.
            mapping (dict): The fields to write.

        Returns:
            int: The number of fields written.
        """
        stored = self.values.setdefault(key, {})
        stored.update(mapping)
        return len(mapping)

    def hmget(self, key, fields):
        """Reads several fields of a hash.

        Args:
            key (str): The hash to read.
            fields (list): The fields to read.

        Returns:
            list: Each field's value, or None where the field is absent.
        """
        stored = self.values.get(key, {})
        replies = []
        for field in fields:
            replies.append(stored.get(field))
        return replies

    def hgetall(self, key):
        """Reads a whole hash.

        Args:
            key (str): The hash to read.

        Returns:
            dict: The hash's fields, empty when the key is absent.
        """
        return dict(self.values.get(key, {}))


class StandInResult:
    """A stand-in for a SQLAlchemy result holding rows already fetched.

    Attributes:
        rows (list): The rows, in order.
    """

    def __init__(self, rows):
        """Holds the rows.

        Args:
            rows (list): The rows, in order.

        Returns:
            None: This method returns nothing.
        """
        self.rows = rows

    def __iter__(self):
        """Iterates over the rows.

        Returns:
            iterator: An iterator over the rows.
        """
        return iter(self.rows)

    def all(self):
        """Returns every row.

        Returns:
            list: The rows.
        """
        return list(self.rows)

    def scalar(self):
        """Returns the first row's single value.

        Returns:
            object: The first row, which for a scalar answer is the value itself, or None when there is no row.
        """
        if not self.rows:
            return None
        return self.rows[0]


class StandInConnection:
    """A stand-in for a SQLAlchemy connection that asks its engine for each statement's rows.

    Attributes:
        engine (StandInEngine): The engine holding the answers.
    """

    def __init__(self, engine):
        """Builds the connection.

        Args:
            engine (StandInEngine): The engine holding the answers.

        Returns:
            None: This method returns nothing.
        """
        self.engine = engine

    def __enter__(self):
        """Enters a `with` block.

        Returns:
            StandInConnection: This connection.
        """
        return self

    def __exit__(self, exception_type, exception, traceback):
        """Leaves a `with` block without suppressing any exception.

        Args:
            exception_type (type | None): The exception's class, if one was raised.
            exception (BaseException | None): The exception, if one was raised.
            traceback (types.TracebackType | None): The traceback, if one was raised.

        Returns:
            bool: Always False.
        """
        return False

    def execute(self, statement, parameters=None):
        """Answers one statement from the engine's rows.

        Args:
            statement (sqlalchemy.sql.elements.TextClause): The statement.
            parameters (dict | None): The bound parameters.

        Returns:
            StandInResult: The rows for the statement.
        """
        return self.engine.answer(str(statement), parameters)


class StandInEngine:
    """A stand-in for a SQLAlchemy engine that answers statements from rows held in memory.

    Each statement's rows are narrowed the way its WHERE clause would narrow them: to the broker it names, and to the tokens or instrument ids it lists.

    Attributes:
        answers (list): Pairs of (text fragment, rows); a statement containing the fragment gets the rows.
        calls (list): Pairs of (text fragment, parameters) for every statement answered.
    """

    def __init__(self):
        """Builds an engine with no answers.

        Returns:
            None: This method returns nothing.
        """
        self.answers = []
        self.calls = []

    def add_answer(self, fragment, rows):
        """Adds the rows a statement containing a fragment should return.

        Args:
            fragment (str): Text that appears in the statement and nowhere in the others.
            rows (list): The rows to return.

        Returns:
            None: This method returns nothing.
        """
        self.answers.append((fragment, rows))

    def connect(self):
        """Opens a connection.

        Returns:
            StandInConnection: A new connection.
        """
        return StandInConnection(self)

    def answer(self, statement_text, parameters):
        """Finds the rows for one statement and records the call.

        Args:
            statement_text (str): The statement's SQL.
            parameters (dict | None): The bound parameters.

        Returns:
            StandInResult: The rows for the statement.

        Raises:
            ValueError: If no fragment matches the statement.
        """
        for fragment, rows in self.answers:
            if fragment in statement_text:
                self.calls.append((fragment, parameters))
                kept = []
                for row in rows:
                    if self.keeps(row, parameters):
                        kept.append(row)
                return StandInResult(kept)
        raise ValueError(f'No stand-in answer for statement: {statement_text}')

    def keeps(self, row, parameters):
        """Says whether a row passes a statement's broker, token and instrument id filters.

        Args:
            row (types.SimpleNamespace): The row.
            parameters (dict | None): The bound parameters.

        Returns:
            bool: True when the row passes every filter the parameters carry.
        """
        if parameters is None:
            return True
        if 'broker' in parameters and row.broker != parameters['broker']:
            return False
        if 'tokens' in parameters and row.broker_token not in parameters['tokens']:
            return False
        if 'instrument_identifiers' in parameters and str(row.instrument_id) not in parameters['instrument_identifiers']:
            return False
        return True


class HandlesForAnOrderExample:
    """Resolves a future's order handles cold, then from a second lookup sharing Redis.

    Attributes:
        mapping_date (datetime.date): The mapping date asked about.
        redis_client (StandInRedis): The Redis stand-in.
        engine (StandInEngine): The Postgres stand-in.
        redis_tier (MappingRedisTier): The Redis tier over the stand-in.
        postgres_tier (MappingPostgresTier): The Postgres tier over the stand-in.
        statistics (dict): The counters the lookups share.
        lookup (OrderHandleLookup): The lookup being shown.
    """

    CRUDEOIL_FUTURE_ID = '4e6a8c0e-2b4d-4f6a-8c0e-2b4d6f8a0c35'

    def __init__(self):
        """Builds the stand-ins, the tiers and the lookup.

        Returns:
            None: This method returns nothing.
        """
        self.mapping_date = datetime.date(2026, 9, 30)
        self.redis_client = StandInRedis()
        self.engine = StandInEngine()
        self.engine.add_answer('SELECT instrument_id, broker, broker_token', [
            self.handle_row(self.CRUDEOIL_FUTURE_ID, 'dhan', '467013', None, '100', '1'),
            self.handle_row(self.CRUDEOIL_FUTURE_ID, 'zerodha', '118235911', 'CRUDEOIL26OCTFUT', '100', '1'),
        ])
        self.redis_tier = MappingRedisTier(MappingRedisConnection(self.redis_client))
        self.postgres_tier = MappingPostgresTier(self.engine)
        self.statistics = {
            'process_hits': 0,
            'redis_hits': 0,
            'postgres_hits': 0,
            'misses': 0,
        }
        self.lookup = OrderHandleLookup(self.redis_tier, self.postgres_tier, self.statistics)

    def handle_row(self, instrument_identifier, broker, broker_token, order_symbol, lot_size, tick_size):
        """Builds one row of `unified.broker_mappings` carrying an order handle.

        Args:
            instrument_identifier (str): The instrument id.
            broker (str): The broker name.
            broker_token (str): The broker's token.
            order_symbol (str | None): The symbol the broker orders by, if any.
            lot_size (str): The lot size, as text.
            tick_size (str): The tick size, as text.

        Returns:
            types.SimpleNamespace: The row.
        """
        return types.SimpleNamespace(
            instrument_id=uuid.UUID(instrument_identifier),
            broker=broker,
            broker_token=broker_token,
            order_symbol=order_symbol,
            lot_size=decimal.Decimal(lot_size),
            tick_size=decimal.Decimal(tick_size),
        )

    def run(self):
        """Resolves the handles twice in this process and once in another, then calls each tier method directly.

        Returns:
            None: This method returns nothing.
        """
        keys = [
            self.CRUDEOIL_FUTURE_ID,
        ]
        handles = self.lookup.resolve(self.mapping_date, keys)
        for broker, handle in handles[self.CRUDEOIL_FUTURE_ID].items():
            print(f'{broker}: {handle}')
        print(f'  counters: {self.statistics}')
        self.lookup.resolve(self.mapping_date, keys)
        print(f'Same process again: {self.statistics}')
        other_worker = OrderHandleLookup(self.redis_tier, self.postgres_tier, self.statistics)
        other_worker.resolve(self.mapping_date, keys)
        print(f'Another worker: {self.statistics}')
        print(f'Postgres queries: {len(self.engine.calls)}')
        stored = self.redis_client.hgetall(self.redis_tier.order_handles_key(self.mapping_date))
        print(f'Stored in Redis: {stored[self.CRUDEOIL_FUTURE_ID]}')
        print(f'empty_entry: {self.lookup.empty_entry()}')
        from_redis = self.lookup.read_from_redis(self.mapping_date, keys)
        print(f'read_from_redis brokers: {sorted(from_redis[self.CRUDEOIL_FUTURE_ID])}')
        from_postgres = self.lookup.read_from_postgres(self.mapping_date, keys)
        print(f'read_from_postgres brokers: {sorted(from_postgres[self.CRUDEOIL_FUTURE_ID])}')
        print(f'write_to_redis: {self.lookup.write_to_redis(self.mapping_date, from_postgres)}')


if __name__ == '__main__':
    HandlesForAnOrderExample().run()

It prints:

dhan: {'broker_token': '467013', 'order_symbol': None, 'lot_size': '100', 'tick_size': '1'}
zerodha: {'broker_token': '118235911', 'order_symbol': 'CRUDEOIL26OCTFUT', 'lot_size': '100', 'tick_size': '1'}
  counters: {'process_hits': 0, 'redis_hits': 0, 'postgres_hits': 1, 'misses': 0}
Same process again: {'process_hits': 1, 'redis_hits': 0, 'postgres_hits': 1, 'misses': 0}
Another worker: {'process_hits': 1, 'redis_hits': 1, 'postgres_hits': 1, 'misses': 0}
Postgres queries: 1
Stored in Redis: {"dhan": {"broker_token": "467013", "order_symbol": null, "lot_size": "100", "tick_size": "1"}, "zerodha": {"broker_token": "118235911", "order_symbol": "CRUDEOIL26OCTFUT", "lot_size": "100", "tick_size": "1"}}
empty_entry: {}
read_from_redis brokers: ['dhan', 'zerodha']
read_from_postgres brokers: ['dhan', 'zerodha']
write_to_redis: True
examples/stock_brokers/instruments/mapping/utilities/cache/OrderHandleLookup/example_2_old_shapes_and_unmapped_instruments.py
"""Shows an order handle left in an older shape being refilled, and an unmapped instrument remembered as empty.

The order handle hash once held a bare broker token per broker rather than the four-field handle it holds now. A Redis still holding a value in that narrower shape must not be decoded into something the order route cannot use, so `MappingRedisTier.decode_order_handles` rejects any value that is not a dictionary of dictionaries, and `OrderHandleLookup.read_from_redis` treats the instrument as absent. The walk then falls through to Postgres and writes the full handle back over the old value.

An instrument that no broker maps on the day, such as a contract that expired yesterday, is a different case. `OrderHandleLookup.empty_entry` is an empty dictionary, so the miss is remembered in this process and the next order on it costs no query.

Two stand-ins replace the data stores: a Redis client that keeps hashes in dictionaries, into which the program writes one value in the old shape by hand, and a Postgres engine that holds the rows of the order handle statement, keeps only the instrument ids a statement asks for, and records every call. Notice the value in Redis before and after, and that the second round asks Postgres nothing.

Run it from the project root:

    python examples/stock_brokers/instruments/mapping/utilities/cache/OrderHandleLookup/example_2_old_shapes_and_unmapped_instruments.py
"""

import datetime
import decimal
import types
import uuid

from stock_brokers.instruments.mapping.utilities.cache import (
    MappingPostgresTier,
    MappingRedisConnection,
    MappingRedisTier,
    OrderHandleLookup,
)


class StandInPipeline:
    """A stand-in for a Redis pipeline that queues commands and runs them on execute.

    Attributes:
        redis_client (StandInRedis): The stand-in client the commands run against.
        queued (list): The queued commands, as (method, arguments) pairs.
    """

    def __init__(self, redis_client):
        """Builds an empty pipeline.

        Args:
            redis_client (StandInRedis): The stand-in client the commands run against.

        Returns:
            None: This method returns nothing.
        """
        self.redis_client = redis_client
        self.queued = []

    def set(self, key, value):
        """Queues a string write.

        Args:
            key (str): The key to write.
            value (str): The value to store.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.set, (key, value)))

    def hset(self, key, mapping):
        """Queues a hash write.

        Args:
            key (str): The hash to write into.
            mapping (dict): The fields to write.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.hset, (key, mapping)))

    def hmget(self, key, fields):
        """Queues a read of several hash fields.

        Args:
            key (str): The hash to read.
            fields (list): The fields to read.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.hmget, (key, fields)))

    def expire(self, key, seconds):
        """Queues an expiry.

        Args:
            key (str): The key to expire.
            seconds (int): How long the key may live.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.expire, (key, seconds)))

    def execute(self):
        """Runs every queued command in order.

        Returns:
            list: Each command's reply, in the order queued.
        """
        replies = []
        for method, arguments in self.queued:
            replies.append(method(*arguments))
        self.queued = []
        return replies


class StandInRedis:
    """A stand-in for a Redis client that keeps strings and hashes in dictionaries.

    Attributes:
        values (dict): Each key's value: a string, or a dict for a hash.
        expiries (dict): Each key's expiry in seconds, as last set.
    """

    def __init__(self):
        """Builds an empty stand-in.

        Returns:
            None: This method returns nothing.
        """
        self.values = {}
        self.expiries = {}

    def pipeline(self, transaction=True):
        """Starts a pipeline.

        Args:
            transaction (bool): Ignored; the stand-in always runs commands in order.

        Returns:
            StandInPipeline: A new pipeline.
        """
        return StandInPipeline(self)

    def get(self, key):
        """Reads a string.

        Args:
            key (str): The key to read.

        Returns:
            str | None: The value, or None when the key is absent.
        """
        return self.values.get(key)

    def set(self, key, value):
        """Writes a string.

        Args:
            key (str): The key to write.
            value (str): The value to store.

        Returns:
            bool: Always True.
        """
        self.values[key] = value
        return True

    def exists(self, key):
        """Counts whether a key exists.

        Args:
            key (str): The key to test.

        Returns:
            int: 1 when the key exists, otherwise 0.
        """
        if key in self.values:
            return 1
        return 0

    def expire(self, key, seconds):
        """Records a key's expiry.

        Args:
            key (str): The key to expire.
            seconds (int): How long the key may live.

        Returns:
            bool: Always True.
        """
        self.expiries[key] = seconds
        return True

    def hset(self, key, mapping):
        """Writes fields into a hash.

        Args:
            key (str): The hash to write into.
            mapping (dict): The fields to write.

        Returns:
            int: The number of fields written.
        """
        stored = self.values.setdefault(key, {})
        stored.update(mapping)
        return len(mapping)

    def hmget(self, key, fields):
        """Reads several fields of a hash.

        Args:
            key (str): The hash to read.
            fields (list): The fields to read.

        Returns:
            list: Each field's value, or None where the field is absent.
        """
        stored = self.values.get(key, {})
        replies = []
        for field in fields:
            replies.append(stored.get(field))
        return replies

    def hgetall(self, key):
        """Reads a whole hash.

        Args:
            key (str): The hash to read.

        Returns:
            dict: The hash's fields, empty when the key is absent.
        """
        return dict(self.values.get(key, {}))


class StandInResult:
    """A stand-in for a SQLAlchemy result holding rows already fetched.

    Attributes:
        rows (list): The rows, in order.
    """

    def __init__(self, rows):
        """Holds the rows.

        Args:
            rows (list): The rows, in order.

        Returns:
            None: This method returns nothing.
        """
        self.rows = rows

    def __iter__(self):
        """Iterates over the rows.

        Returns:
            iterator: An iterator over the rows.
        """
        return iter(self.rows)

    def all(self):
        """Returns every row.

        Returns:
            list: The rows.
        """
        return list(self.rows)

    def scalar(self):
        """Returns the first row's single value.

        Returns:
            object: The first row, which for a scalar answer is the value itself, or None when there is no row.
        """
        if not self.rows:
            return None
        return self.rows[0]


class StandInConnection:
    """A stand-in for a SQLAlchemy connection that asks its engine for each statement's rows.

    Attributes:
        engine (StandInEngine): The engine holding the answers.
    """

    def __init__(self, engine):
        """Builds the connection.

        Args:
            engine (StandInEngine): The engine holding the answers.

        Returns:
            None: This method returns nothing.
        """
        self.engine = engine

    def __enter__(self):
        """Enters a `with` block.

        Returns:
            StandInConnection: This connection.
        """
        return self

    def __exit__(self, exception_type, exception, traceback):
        """Leaves a `with` block without suppressing any exception.

        Args:
            exception_type (type | None): The exception's class, if one was raised.
            exception (BaseException | None): The exception, if one was raised.
            traceback (types.TracebackType | None): The traceback, if one was raised.

        Returns:
            bool: Always False.
        """
        return False

    def execute(self, statement, parameters=None):
        """Answers one statement from the engine's rows.

        Args:
            statement (sqlalchemy.sql.elements.TextClause): The statement.
            parameters (dict | None): The bound parameters.

        Returns:
            StandInResult: The rows for the statement.
        """
        return self.engine.answer(str(statement), parameters)


class StandInEngine:
    """A stand-in for a SQLAlchemy engine that answers statements from rows held in memory.

    Each statement's rows are narrowed the way its WHERE clause would narrow them: to the broker it names, and to the tokens or instrument ids it lists.

    Attributes:
        answers (list): Pairs of (text fragment, rows); a statement containing the fragment gets the rows.
        calls (list): Pairs of (text fragment, parameters) for every statement answered.
    """

    def __init__(self):
        """Builds an engine with no answers.

        Returns:
            None: This method returns nothing.
        """
        self.answers = []
        self.calls = []

    def add_answer(self, fragment, rows):
        """Adds the rows a statement containing a fragment should return.

        Args:
            fragment (str): Text that appears in the statement and nowhere in the others.
            rows (list): The rows to return.

        Returns:
            None: This method returns nothing.
        """
        self.answers.append((fragment, rows))

    def connect(self):
        """Opens a connection.

        Returns:
            StandInConnection: A new connection.
        """
        return StandInConnection(self)

    def answer(self, statement_text, parameters):
        """Finds the rows for one statement and records the call.

        Args:
            statement_text (str): The statement's SQL.
            parameters (dict | None): The bound parameters.

        Returns:
            StandInResult: The rows for the statement.

        Raises:
            ValueError: If no fragment matches the statement.
        """
        for fragment, rows in self.answers:
            if fragment in statement_text:
                self.calls.append((fragment, parameters))
                kept = []
                for row in rows:
                    if self.keeps(row, parameters):
                        kept.append(row)
                return StandInResult(kept)
        raise ValueError(f'No stand-in answer for statement: {statement_text}')

    def keeps(self, row, parameters):
        """Says whether a row passes a statement's broker, token and instrument id filters.

        Args:
            row (types.SimpleNamespace): The row.
            parameters (dict | None): The bound parameters.

        Returns:
            bool: True when the row passes every filter the parameters carry.
        """
        if parameters is None:
            return True
        if 'broker' in parameters and row.broker != parameters['broker']:
            return False
        if 'tokens' in parameters and row.broker_token not in parameters['tokens']:
            return False
        if 'instrument_identifiers' in parameters and str(row.instrument_id) not in parameters['instrument_identifiers']:
            return False
        return True


class OldShapesExample:
    """Resolves one instrument held in an old shape and one no broker maps, twice.

    Attributes:
        mapping_date (datetime.date): The mapping date asked about.
        redis_client (StandInRedis): The Redis stand-in.
        engine (StandInEngine): The Postgres stand-in.
        redis_tier (MappingRedisTier): The Redis tier over the stand-in.
        statistics (dict): The counters the lookup counts into.
        lookup (OrderHandleLookup): The lookup being shown.
    """

    SBIN_ID = '1d3f5b7d-9f1b-4d3f-a5b7-d9f1b3d5f746'
    EXPIRED_FUTURE_ID = '3a5c7e9a-1c3e-4a5c-b7e9-a1c3e5a7c957'

    def __init__(self):
        """Builds the stand-ins, the tiers and the lookup, and leaves an old-shaped value in Redis.

        Returns:
            None: This method returns nothing.
        """
        self.mapping_date = datetime.date(2026, 9, 30)
        self.redis_client = StandInRedis()
        self.engine = StandInEngine()
        self.engine.add_answer('SELECT instrument_id, broker, broker_token', [
            self.handle_row(self.SBIN_ID, 'shoonya', '3045', 'SBIN-EQ', '1', '0.05'),
            self.handle_row(self.SBIN_ID, 'zerodha', '779521', 'SBIN', '1', '0.05'),
        ])
        self.redis_tier = MappingRedisTier(MappingRedisConnection(self.redis_client))
        self.statistics = {
            'process_hits': 0,
            'redis_hits': 0,
            'postgres_hits': 0,
            'misses': 0,
        }
        self.lookup = OrderHandleLookup(self.redis_tier, MappingPostgresTier(self.engine), self.statistics)
        old_shape = {
            self.SBIN_ID: '{"shoonya": "3045", "zerodha": "779521"}',
        }
        self.redis_client.hset(self.redis_tier.order_handles_key(self.mapping_date), old_shape)

    def handle_row(self, instrument_identifier, broker, broker_token, order_symbol, lot_size, tick_size):
        """Builds one row of `unified.broker_mappings` carrying an order handle.

        Args:
            instrument_identifier (str): The instrument id.
            broker (str): The broker name.
            broker_token (str): The broker's token.
            order_symbol (str | None): The symbol the broker orders by, if any.
            lot_size (str): The lot size, as text.
            tick_size (str): The tick size, as text.

        Returns:
            types.SimpleNamespace: The row.
        """
        return types.SimpleNamespace(
            instrument_id=uuid.UUID(instrument_identifier),
            broker=broker,
            broker_token=broker_token,
            order_symbol=order_symbol,
            lot_size=decimal.Decimal(lot_size),
            tick_size=decimal.Decimal(tick_size),
        )

    def stored_value(self):
        """Reads SBIN's raw value from the order handle hash.

        Returns:
            str | None: The stored JSON text.
        """
        return self.redis_client.hgetall(self.redis_tier.order_handles_key(self.mapping_date)).get(self.SBIN_ID)

    def run(self):
        """Resolves both instruments twice and prints the Redis value, the answers and the counters.

        Returns:
            None: This method returns nothing.
        """
        keys = [
            self.SBIN_ID,
            self.EXPIRED_FUTURE_ID,
        ]
        print(f'Redis holds, in the old shape: {self.stored_value()}')
        print(f'read_from_redis finds: {self.lookup.read_from_redis(self.mapping_date, keys)}')
        answer = self.lookup.resolve(self.mapping_date, keys)
        print(f'SBIN handles: {answer[self.SBIN_ID]}')
        print(f'Expired future: {answer[self.EXPIRED_FUTURE_ID]}')
        print(f'  counters: {self.statistics}')
        print(f'Redis now holds: {self.stored_value()}')
        self.lookup.resolve(self.mapping_date, keys)
        print(f'Second round counters: {self.statistics}')
        print(f'Postgres queries: {len(self.engine.calls)}')
        print(f'empty_entry: {self.lookup.empty_entry()}')
        refilled = self.lookup.read_from_postgres(self.mapping_date, keys)
        print(f'read_from_postgres finds: {sorted(refilled)}')
        print(f'write_to_redis: {self.lookup.write_to_redis(self.mapping_date, refilled)}')


if __name__ == '__main__':
    OldShapesExample().run()

It prints:

Redis holds, in the old shape: {"shoonya": "3045", "zerodha": "779521"}
read_from_redis finds: {}
SBIN handles: {'shoonya': {'broker_token': '3045', 'order_symbol': 'SBIN-EQ', 'lot_size': '1', 'tick_size': '0.05'}, 'zerodha': {'broker_token': '779521', 'order_symbol': 'SBIN', 'lot_size': '1', 'tick_size': '0.05'}}
Expired future: {}
  counters: {'process_hits': 0, 'redis_hits': 0, 'postgres_hits': 1, 'misses': 1}
Redis now holds: {"shoonya": {"broker_token": "3045", "order_symbol": "SBIN-EQ", "lot_size": "1", "tick_size": "0.05"}, "zerodha": {"broker_token": "779521", "order_symbol": "SBIN", "lot_size": "1", "tick_size": "0.05"}}
Second round counters: {'process_hits': 2, 'redis_hits': 0, 'postgres_hits': 1, 'misses': 1}
Postgres queries: 1
empty_entry: {}
read_from_postgres finds: ['1d3f5b7d-9f1b-4d3f-a5b7-d9f1b3d5f746']
write_to_redis: True

Build the lookup around the two lower tiers and the shared counters.

Parameters:

Name Type Description Default
redis_tier MappingRedisTier

The Redis hashes to read and write.

 required
postgres_tier MappingPostgresTier

The statements to fall through to.

 required
statistics dict

The counters to count into.

 required

Returns:

Type Description
None

This function returns nothing.

Example programs

examples/stock_brokers/instruments/mapping/utilities/cache/ThreeTierLookup/example_1_a_lot_size_lookup.py
"""Builds a small lookup on top of `ThreeTierLookup` and follows one key from Postgres into Redis and into process memory.

`ThreeTierLookup` holds the walk every mapping cache question shares: look in this process's own dictionary, then in Redis, then in Postgres, and copy each answer into the tiers above the one that had it. A subclass supplies only three methods, `read_from_redis`, `read_from_postgres` and `write_to_redis`. The three lookups in `cache.py` are such subclasses; this program writes a fourth, which answers an instrument id with its lot size, so the base class's own behaviour can be seen without the real tiers' detail.

The "Redis" and "Postgres" tiers here are two small stand-in classes holding dictionaries, because the base class never looks inside its tiers and only the subclass talks to them. Notice how the shared `statistics` counters move: the first `resolve` is answered by Postgres, the second by this process, and a second lookup, standing in for another process, is answered by Redis because the first one wrote its answer there. `remember`, `known`, `entry` and `reset` work on the process tier alone and never touch the other two.

Run it from the project root:

    python examples/stock_brokers/instruments/mapping/utilities/cache/ThreeTierLookup/example_1_a_lot_size_lookup.py
"""

import datetime

from stock_brokers.instruments.mapping.utilities.cache import (
    ThreeTierLookup,
)


class DictionaryTier:
    """A stand-in for a Redis or Postgres tier that holds lot sizes in a dictionary.

    Attributes:
        lot_sizes (dict): Each instrument id's lot size.
        reads (int): How many times the tier was read.
    """

    def __init__(self, lot_sizes):
        """Builds the tier.

        Args:
            lot_sizes (dict): Each instrument id's lot size.

        Returns:
            None: This method returns nothing.
        """
        self.lot_sizes = lot_sizes
        self.reads = 0

    def read(self, keys):
        """Reads the lot sizes of the keys this tier holds.

        Args:
            keys (list): The instrument ids to read.

        Returns:
            dict: Each held instrument id to an entry with its lot size.
        """
        self.reads += 1
        found = {}
        for key in keys:
            if key in self.lot_sizes:
                found[key] = {
                    'lot_size': self.lot_sizes[key],
                }
        return found

    def write(self, entries):
        """Stores entries' lot sizes.

        Args:
            entries (dict): Each instrument id to an entry with its lot size.

        Returns:
            bool: Always True.
        """
        for key, entry in entries.items():
            self.lot_sizes[key] = entry['lot_size']
        return True


class LotSizeLookup(ThreeTierLookup):
    """Instrument id to that instrument's lot size, through the three tiers."""

    def read_from_redis(self, mapping_date, keys):
        """Reads lot sizes out of the Redis stand-in.

        Args:
            mapping_date (datetime.date): The mapping date to read.
            keys (list): The instrument ids to read.

        Returns:
            dict: Each held instrument id to its entry.
        """
        return self.redis_tier.read(keys)

    def read_from_postgres(self, mapping_date, keys):
        """Reads lot sizes out of the Postgres stand-in.

        Args:
            mapping_date (datetime.date): The mapping date to read.
            keys (list): The instrument ids to read.

        Returns:
            dict: Each held instrument id to its entry.
        """
        return self.postgres_tier.read(keys)

    def write_to_redis(self, mapping_date, entries):
        """Writes lot sizes into the Redis stand-in.

        Args:
            mapping_date (datetime.date): The mapping date the entries belong to.
            entries (dict): Each instrument id to its entry.

        Returns:
            bool: True when the write went through.
        """
        return self.redis_tier.write(entries)


class LotSizeLookupExample:
    """Resolves lot sizes through a first and a second lookup sharing one Redis stand-in.

    Attributes:
        mapping_date (datetime.date): The mapping date asked about.
        redis_tier (DictionaryTier): The Redis stand-in, empty at first.
        postgres_tier (DictionaryTier): The Postgres stand-in, holding every lot size.
        statistics (dict): The counters the lookups share.
        lookup (LotSizeLookup): The first process's lookup.
    """

    def __init__(self):
        """Builds the tiers, the counters and the first lookup.

        Returns:
            None: This method returns nothing.
        """
        self.mapping_date = datetime.date(2026, 9, 30)
        self.redis_tier = DictionaryTier({})
        self.postgres_tier = DictionaryTier({
            'nifty-future': '75',
            'banknifty-future': '35',
            'crudeoil-future': '100',
        })
        self.statistics = {
            'process_hits': 0,
            'redis_hits': 0,
            'postgres_hits': 0,
            'misses': 0,
        }
        self.lookup = LotSizeLookup(self.redis_tier, self.postgres_tier, self.statistics)

    def run(self):
        """Resolves the same keys cold, warm and from a second lookup, then uses the process tier directly.

        Returns:
            None: This method returns nothing.
        """
        keys = [
            'nifty-future',
            'banknifty-future',
        ]
        print(f'First resolve: {self.lookup.resolve(self.mapping_date, keys)}')
        print(f'  counters: {self.statistics}')
        print(f'Second resolve: {self.lookup.resolve(self.mapping_date, keys)}')
        print(f'  counters: {self.statistics}')
        print(f'Redis now holds: {self.redis_tier.lot_sizes}')
        second_process = LotSizeLookup(self.redis_tier, self.postgres_tier, self.statistics)
        print(f'Second process resolves: {second_process.resolve(self.mapping_date, keys)}')
        print(f'  counters: {self.statistics}')
        print(f'Postgres was read {self.postgres_tier.reads} time')
        print(f'Known before remember: {self.lookup.known("crudeoil-future")}')
        crude_entry = {
            'lot_size': '100',
        }
        self.lookup.remember('crudeoil-future', crude_entry)
        print(f'Known after remember: {self.lookup.known("crudeoil-future")}')
        print(f'Entry held for it: {self.lookup.entry("crudeoil-future")}')
        print(f'Entry for a key never seen: {self.lookup.entry("goldm-future")}')
        print(f'What a miss is remembered as: {self.lookup.empty_entry()}')
        crude_only = [
            'crudeoil-future',
        ]
        print(f'Reading Redis directly: {self.lookup.read_from_redis(self.mapping_date, crude_only)}')
        print(f'Reading Postgres directly: {self.lookup.read_from_postgres(self.mapping_date, crude_only)}')
        crude_entries = {
            'crudeoil-future': crude_entry,
        }
        print(f'Writing Redis directly: {self.lookup.write_to_redis(self.mapping_date, crude_entries)}')
        self.lookup.reset()
        print(f'Known after reset: {self.lookup.known("nifty-future")}')


if __name__ == '__main__':
    LotSizeLookupExample().run()

It prints:

First resolve: {'nifty-future': {'lot_size': '75'}, 'banknifty-future': {'lot_size': '35'}}
  counters: {'process_hits': 0, 'redis_hits': 0, 'postgres_hits': 2, 'misses': 0}
Second resolve: {'nifty-future': {'lot_size': '75'}, 'banknifty-future': {'lot_size': '35'}}
  counters: {'process_hits': 2, 'redis_hits': 0, 'postgres_hits': 2, 'misses': 0}
Redis now holds: {'nifty-future': '75', 'banknifty-future': '35'}
Second process resolves: {'nifty-future': {'lot_size': '75'}, 'banknifty-future': {'lot_size': '35'}}
  counters: {'process_hits': 2, 'redis_hits': 2, 'postgres_hits': 2, 'misses': 0}
Postgres was read 1 time
Known before remember: False
Known after remember: True
Entry held for it: {'lot_size': '100'}
Entry for a key never seen: None
What a miss is remembered as: None
Reading Redis directly: {}
Reading Postgres directly: {'crudeoil-future': {'lot_size': '100'}}
Writing Redis directly: True
Known after reset: False
examples/stock_brokers/instruments/mapping/utilities/cache/ThreeTierLookup/example_2_remembering_misses.py
"""Shows what the base class does on its own, and how `empty_entry` decides whether a miss is asked about again.

`ThreeTierLookup` is meant to be subclassed. Used directly, its three tier methods raise `NotImplementedError` naming the class, while an empty batch of keys is answered without reaching any tier at all.

The more interesting choice a subclass makes is `empty_entry`. When Postgres has no answer for a key, the walk asks `empty_entry()` what to remember. The base answer, None, remembers nothing, so the next look-up of that key goes all the way to Postgres again; the cache's identity lookup works this way, because an unknown instrument id is a caller's mistake that tomorrow's master may fix. A subclass that returns an empty value instead makes the miss a fact for the rest of the day; the cache's token lookup works this way, because a token a broker does not carry is asked about on every tick.

This program builds two small subclasses that differ only in `empty_entry` and asks each about the same unknown key three times, against a stand-in Postgres that counts its reads. Notice that the first asks Postgres three times and the second once.

Run it from the project root:

    python examples/stock_brokers/instruments/mapping/utilities/cache/ThreeTierLookup/example_2_remembering_misses.py
"""

import datetime

from stock_brokers.instruments.mapping.utilities.cache import (
    ThreeTierLookup,
)


class CountingEmptyTier:
    """A stand-in tier that holds nothing and counts how often it is read.

    Attributes:
        reads (int): How many times the tier was read.
    """

    def __init__(self):
        """Builds the tier with no reads counted.

        Returns:
            None: This method returns nothing.
        """
        self.reads = 0

    def read(self, keys):
        """Counts one read and finds nothing.

        Args:
            keys (list): The keys asked about.

        Returns:
            dict: Always empty.
        """
        self.reads += 1
        return {}


class ForgettingLookup(ThreeTierLookup):
    """A lookup that keeps the base class's `empty_entry`, so it never remembers a miss."""

    def read_from_redis(self, mapping_date, keys):
        """Reads the Redis stand-in.

        Args:
            mapping_date (datetime.date): The mapping date to read.
            keys (list): The keys to read.

        Returns:
            dict: What the stand-in held.
        """
        return self.redis_tier.read(keys)

    def read_from_postgres(self, mapping_date, keys):
        """Reads the Postgres stand-in.

        Args:
            mapping_date (datetime.date): The mapping date to read.
            keys (list): The keys to read.

        Returns:
            dict: What the stand-in held.
        """
        return self.postgres_tier.read(keys)

    def write_to_redis(self, mapping_date, entries):
        """Accepts entries without storing them.

        Args:
            mapping_date (datetime.date): The mapping date the entries belong to.
            entries (dict): The entries to write.

        Returns:
            bool: Always True.
        """
        return True


class RememberingLookup(ForgettingLookup):
    """A lookup that remembers a miss as an empty list for the rest of the day."""

    def empty_entry(self):
        """What to remember for a key nothing maps.

        Returns:
            list: A new empty list.
        """
        return []


class RememberingMissesExample:
    """Calls the base class directly, then compares a forgetting and a remembering lookup.

    Attributes:
        mapping_date (datetime.date): The mapping date asked about.
        statistics (dict): The counters the lookups count into.
    """

    def __init__(self):
        """Builds the date and the counters.

        Returns:
            None: This method returns nothing.
        """
        self.mapping_date = datetime.date(2026, 9, 30)
        self.statistics = {
            'process_hits': 0,
            'redis_hits': 0,
            'postgres_hits': 0,
            'misses': 0,
        }

    def show_base_class(self):
        """Calls each method of the base class itself and prints what happens.

        Returns:
            None: This method returns nothing.
        """
        base = ThreeTierLookup(None, None, self.statistics)
        print(f'Base resolve of no keys: {base.resolve(self.mapping_date, [])}')
        print(f'Base empty_entry: {base.empty_entry()}')
        keys = [
            '999999',
        ]
        try:
            base.resolve(self.mapping_date, keys)
        except NotImplementedError as error:
            print(f'Base resolve of a key: NotImplementedError: {error}')
        try:
            base.read_from_redis(self.mapping_date, keys)
        except NotImplementedError as error:
            print(f'Base read_from_redis: NotImplementedError: {error}')
        try:
            base.read_from_postgres(self.mapping_date, keys)
        except NotImplementedError as error:
            print(f'Base read_from_postgres: NotImplementedError: {error}')
        try:
            base.write_to_redis(self.mapping_date, {})
        except NotImplementedError as error:
            print(f'Base write_to_redis: NotImplementedError: {error}')

    def ask_three_times(self, lookup, postgres_tier):
        """Asks one lookup about an unknown key three times and prints the result.

        Args:
            lookup (ThreeTierLookup): The lookup to ask.
            postgres_tier (CountingEmptyTier): The Postgres stand-in behind it.

        Returns:
            None: This method returns nothing.
        """
        keys = [
            '999999',
        ]
        answers = []
        for attempt in range(3):
            answers.append(lookup.resolve(self.mapping_date, keys))
        name = type(lookup).__name__
        print(f'{name}: answers {answers}, Postgres read {postgres_tier.reads} times, remembered: {lookup.known("999999")}')

    def run(self):
        """Shows the base class, then the two lookups side by side.

        Returns:
            None: This method returns nothing.
        """
        self.show_base_class()
        forgetting_postgres = CountingEmptyTier()
        forgetting = ForgettingLookup(CountingEmptyTier(), forgetting_postgres, self.statistics)
        self.ask_three_times(forgetting, forgetting_postgres)
        remembering_postgres = CountingEmptyTier()
        remembering = RememberingLookup(CountingEmptyTier(), remembering_postgres, self.statistics)
        self.ask_three_times(remembering, remembering_postgres)
        print(f'Counters: {self.statistics}')


if __name__ == '__main__':
    RememberingMissesExample().run()

It prints:

Base resolve of no keys: {}
Base empty_entry: None
Base resolve of a key: NotImplementedError: ThreeTierLookup must implement read_from_redis()
Base read_from_redis: NotImplementedError: ThreeTierLookup must implement read_from_redis()
Base read_from_postgres: NotImplementedError: ThreeTierLookup must implement read_from_postgres()
Base write_to_redis: NotImplementedError: ThreeTierLookup must implement write_to_redis()
ForgettingLookup: answers [{}, {}, {}], Postgres read 3 times, remembered: False
RememberingLookup: answers [{'999999': []}, {'999999': []}, {'999999': []}], Postgres read 1 times, remembered: True
Counters: {'process_hits': 2, 'redis_hits': 0, 'postgres_hits': 0, 'misses': 4}

Methods:

Name Description
empty_entry

What to remember for an instrument no broker maps on the date.
read_from_redis

Read order handles out of the order handle hash.
read_from_postgres

Read order handles out of unified.broker_mappings.
write_to_redis

Write order handles back into the order handle hash.
resolve

Answer for a batch of keys, trying each tier in turn and filling in the ones above.
remember

Put one entry into this process's memory without going to any other tier.
known

Whether this process already holds an entry for one key.
entry

The entry this process holds for one key, without reaching any other tier.
reset

Drop everything this lookup holds in process memory.
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1637
1638
1639
1640
1641
1642
1643
1644
1645
1646
1647
1648
1649
1650
1651
1652
1653
def __init__(self, redis_tier, postgres_tier, statistics):
    """
    Build the lookup around the two lower tiers and the shared counters.

    Args:
        redis_tier (MappingRedisTier): The Redis hashes to read and write.
        postgres_tier (MappingPostgresTier): The statements to fall through to.
        statistics (dict): The counters to count into.

    Returns:
        None: This function returns nothing.
    """
    self.redis_tier = redis_tier
    self.postgres_tier = postgres_tier
    self.statistics = statistics

    self._entries = {}

empty_entry 

empty_entry()

What to remember for an instrument no broker maps on the date.

Returns:

Type Description
dict

A new empty dictionary.

Example usage

From example_1_handles_for_an_order.py
print(f'empty_entry: {self.lookup.empty_entry()}')
From example_2_old_shapes_and_unmapped_instruments.py
print(f'empty_entry: {self.lookup.empty_entry()}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1996
1997
1998
1999
2000
2001
2002
2003
def empty_entry(self):
    """
    What to remember for an instrument no broker maps on the date.

    Returns:
        dict: A new empty dictionary.
    """
    return {}

read_from_redis 

read_from_redis(mapping_date, keys)

Read order handles out of the order handle hash.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required
keys list[str]

The instrument ids to read.

 required

Returns:

Type Description
dict

Mapping of instrument id to a dict of broker name to handle.

Example usage

From example_1_handles_for_an_order.py
from_redis = self.lookup.read_from_redis(self.mapping_date, keys)
From example_2_old_shapes_and_unmapped_instruments.py
print(f'read_from_redis finds: {self.lookup.read_from_redis(self.mapping_date, keys)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
def read_from_redis(self, mapping_date, keys):
    """
    Read order handles out of the order handle hash.

    Args:
        mapping_date (datetime.date): The mapping date to read.
        keys (list[str]): The instrument ids to read.

    Returns:
        dict: Mapping of instrument id to a dict of broker name to handle.
    """
    return self.redis_tier.read_order_handles(mapping_date, keys)

read_from_postgres 

read_from_postgres(mapping_date, keys)

Read order handles out of unified.broker_mappings.

Parameters:

Name Type Description Default
mapping_date date

The mapping date to read.

 required
keys list[str]

The instrument ids to read.

 required

Returns:

Type Description
dict

Mapping of instrument id to a dict of broker name to handle.

Example usage

From example_1_handles_for_an_order.py
from_postgres = self.lookup.read_from_postgres(self.mapping_date, keys)
From example_2_old_shapes_and_unmapped_instruments.py
refilled = self.lookup.read_from_postgres(self.mapping_date, keys)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
2018
2019
2020
2021
2022
2023
2024
2025
2026
2027
2028
2029
def read_from_postgres(self, mapping_date, keys):
    """
    Read order handles out of unified.broker_mappings.

    Args:
        mapping_date (datetime.date): The mapping date to read.
        keys (list[str]): The instrument ids to read.

    Returns:
        dict: Mapping of instrument id to a dict of broker name to handle.
    """
    return self.postgres_tier.read_order_handles(mapping_date, keys)

write_to_redis 

write_to_redis(mapping_date, entries)

Write order handles back into the order handle hash.

Parameters:

Name Type Description Default
mapping_date date

The mapping date the answers belong to.

 required
entries dict

Mapping of instrument id to a dict of broker name to handle.

 required

Returns:

Type Description
bool

True when the write went through, False when Redis was not usable.

Example usage

From example_1_handles_for_an_order.py
print(f'write_to_redis: {self.lookup.write_to_redis(self.mapping_date, from_postgres)}')
From example_2_old_shapes_and_unmapped_instruments.py
print(f'write_to_redis: {self.lookup.write_to_redis(self.mapping_date, refilled)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
2031
2032
2033
2034
2035
2036
2037
2038
2039
2040
2041
2042
def write_to_redis(self, mapping_date, entries):
    """
    Write order handles back into the order handle hash.

    Args:
        mapping_date (datetime.date): The mapping date the answers belong to.
        entries (dict): Mapping of instrument id to a dict of broker name to handle.

    Returns:
        bool: True when the write went through, False when Redis was not usable.
    """
    return self.redis_tier.write_order_handles(mapping_date, entries)

resolve 

resolve(mapping_date, keys)

Answer for a batch of keys, trying each tier in turn and filling in the ones above.

A key that nothing maps is remembered as an empty entry when the subclass says so, which stops a token nothing carries being asked about on every call. A subclass whose empty_entry is None does not remember misses at all.

Parameters:

Name Type Description Default
mapping_date date

The mapping date being answered for.

 required
keys list[str]

The keys to answer for.

 required

Returns:

Type Description
dict

Mapping of key to its entry, for every key that had one. Keys nothing maps are absent, unless the subclass remembers empty entries, in which case they carry one.

Example usage

From example_1_a_lot_size_lookup.py
print(f'First resolve: {self.lookup.resolve(self.mapping_date, keys)}')
From example_1_a_lot_size_lookup.py
print(f'Second resolve: {self.lookup.resolve(self.mapping_date, keys)}')
From example_1_a_lot_size_lookup.py
print(f'Second process resolves: {second_process.resolve(self.mapping_date, keys)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1655
1656
1657
1658
1659
1660
1661
1662
1663
1664
1665
1666
1667
1668
1669
1670
1671
1672
1673
1674
1675
1676
1677
1678
1679
1680
1681
1682
1683
1684
1685
1686
1687
1688
1689
1690
1691
1692
1693
1694
1695
1696
1697
1698
1699
1700
1701
1702
1703
1704
1705
1706
1707
1708
1709
def resolve(self, mapping_date, keys):
    """
    Answer for a batch of keys, trying each tier in turn and filling in the ones above.

    A key that nothing maps is remembered as an empty entry when the subclass says so, which stops a token nothing carries being asked about on every call. A subclass whose empty_entry is None does not remember misses at all.

    Args:
        mapping_date (datetime.date): The mapping date being answered for.
        keys (list[str]): The keys to answer for.

    Returns:
        dict: Mapping of key to its entry, for every key that had one. Keys nothing maps are absent, unless the subclass remembers empty entries, in which case they carry one.
    """
    if not keys:
        return {}

    resolved = {}
    outstanding = []
    for key in keys:
        entry = self._entries.get(key)
        if entry is None:
            outstanding.append(key)
            continue
        self.statistics["process_hits"] += 1
        resolved[key] = entry

    if outstanding:
        from_redis = self.read_from_redis(mapping_date, outstanding)
        still_outstanding = []
        for key in outstanding:
            if key not in from_redis:
                still_outstanding.append(key)
                continue
            self.statistics["redis_hits"] += 1
            self._entries[key] = from_redis[key]
            resolved[key] = from_redis[key]
        outstanding = still_outstanding

    if outstanding:
        from_postgres = self.read_from_postgres(mapping_date, outstanding)
        for key in outstanding:
            entry = from_postgres.get(key)
            if entry is None:
                self.statistics["misses"] += 1
                empty = self.empty_entry()
                if empty is not None:
                    self._entries[key] = empty
                    resolved[key] = empty
                continue
            self.statistics["postgres_hits"] += 1
            self._entries[key] = entry
            resolved[key] = entry
        self.write_to_redis(mapping_date, from_postgres)

    return resolved

remember 

remember(key, entry)

Put one entry into this process's memory without going to any other tier.

This exists so that a lookup filling itself can also fill a sibling, which is how a token's candidate identities land in the identity lookup rather than being read again from Redis.

Parameters:

Name Type Description Default
key str

The key to remember the entry under.

 required
entry dict

The entry to remember.

 required

Returns:

Type Description
None

This function returns nothing.

Example usage

From example_1_a_lot_size_lookup.py
self.lookup.remember('crudeoil-future', crude_entry)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1711
1712
1713
1714
1715
1716
1717
1718
1719
1720
1721
1722
1723
1724
def remember(self, key, entry):
    """
    Put one entry into this process's memory without going to any other tier.

    This exists so that a lookup filling itself can also fill a sibling, which is how a token's candidate identities land in the identity lookup rather than being read again from Redis.

    Args:
        key (str): The key to remember the entry under.
        entry (dict): The entry to remember.

    Returns:
        None: This function returns nothing.
    """
    self._entries[key] = entry

known 

known(key)

Whether this process already holds an entry for one key.

Parameters:

Name Type Description Default
key str

The key to test.

 required

Returns:

Type Description
bool

True when the entry is in this process's memory.

Example usage

From example_1_a_lot_size_lookup.py
print(f'Known before remember: {self.lookup.known("crudeoil-future")}')
From example_1_a_lot_size_lookup.py
print(f'Known after remember: {self.lookup.known("crudeoil-future")}')
From example_1_a_lot_size_lookup.py
print(f'Known after reset: {self.lookup.known("nifty-future")}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1726
1727
1728
1729
1730
1731
1732
1733
1734
1735
1736
def known(self, key):
    """
    Whether this process already holds an entry for one key.

    Args:
        key (str): The key to test.

    Returns:
        bool: True when the entry is in this process's memory.
    """
    return key in self._entries

entry 

entry(key)

The entry this process holds for one key, without reaching any other tier.

The counterpart of remember, for a sibling lookup assembling an answer out of entries this one already has.

Parameters:

Name Type Description Default
key str

The key to read.

 required

Returns:

Type Description
dict | None

The entry, or None when this process does not hold one.

Example usage

From example_1_a_lot_size_lookup.py
print(f'Entry held for it: {self.lookup.entry("crudeoil-future")}')
From example_1_a_lot_size_lookup.py
print(f'Entry for a key never seen: {self.lookup.entry("goldm-future")}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1738
1739
1740
1741
1742
1743
1744
1745
1746
1747
1748
1749
1750
def entry(self, key):
    """
    The entry this process holds for one key, without reaching any other tier.

    The counterpart of remember, for a sibling lookup assembling an answer out of entries this one already has.

    Args:
        key (str): The key to read.

    Returns:
        dict | None: The entry, or None when this process does not hold one.
    """
    return self._entries.get(key)

reset 

reset()

Drop everything this lookup holds in process memory.

Returns:

Type Description
None

This function returns nothing.

Example usage

From example_1_a_lot_size_lookup.py
self.lookup.reset()
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
1752
1753
1754
1755
1756
1757
1758
1759
def reset(self):
    """
    Drop everything this lookup holds in process memory.

    Returns:
        None: This function returns nothing.
    """
    self._entries.clear()

MappingCache 

MappingCache(engine=None, redis_connection=None)

A three tier cache in front of the mapped instrument tables, holding its own engine.

One of these belongs to a process, not to a call. The memory tier is the whole point of it, and two of them in one process would hold two copies of everything and halve the hit rate, so a caller builds one and passes it down. The engine is available as an attribute for the callers that need one of their own.

Attributes:

Name Type Description
engine Engine

The SQLAlchemy engine over TimescaleDB.
redis_tier MappingRedisTier

The Redis hashes, read and written through one class.
postgres_tier MappingPostgresTier

Every statement, in one class.
statistics dict

Counts with keys "process_hits", "redis_hits", "postgres_hits" and "misses", shared by every lookup.

Build the cache and its tiers.

Parameters:

Name Type Description Default
engine Engine

A SQLAlchemy engine over TimescaleDB. None builds one from the environment.

 None
redis_connection MappingRedisConnection | None

The Redis connection to use. None builds one.

 None

Returns:

Type Description
None

This function returns nothing.

Example programs

examples/stock_brokers/instruments/mapping/utilities/cache/MappingCache/example_1_resolving_for_an_order.py
"""Resolves broker tokens and instrument ids through a `MappingCache`, the object a process holds for every mapping question.

A `MappingCache` is built once per process and passed down. It reads the current mapping date from Redis, and then answers five questions through three shared lookups: which instrument a broker token stands for (`identities_for_tokens`), every instrument it could stand for (`candidates_for_tokens`), what an instrument id is (`identities_for_instruments`), how each broker wants an order on it named (`order_handles_for_instruments`), and just each broker's token for it (`tokens_for_instruments`). Every answer found in Postgres is copied into Redis and into this process, and `cache_statistics()` counts which tier answered.

Two stand-ins replace the data stores, passed in through the cache's `engine` and `redis_connection` arguments. The Redis stand-in keeps strings and hashes in dictionaries and already holds today's mapping date, as it would after the morning warm. The Postgres engine stand-in holds the rows of the three statements the cache runs, keeps only the broker, tokens and instrument ids each statement asks for, and records every call.

Notice that Dhan's security id 1594 carries two instruments in this program's made-up rows: without a segment filter the first in tie-break order wins, with `nse_equities` the NSE listing wins, and `candidates_for_tokens` shows both. The id look-up afterwards is a process hit, because the token look-up already learned the identity. Tokens given as integers are treated as text, and an as-of date later than the mapping date, such as a holiday, is answered from the current mapping.

Run it from the project root:

    python examples/stock_brokers/instruments/mapping/utilities/cache/MappingCache/example_1_resolving_for_an_order.py
"""

import datetime
import decimal
import types
import uuid

from stock_brokers.instruments.mapping.utilities.cache import (
    MappingCache,
    MappingRedisConnection,
)


class StandInPipeline:
    """A stand-in for a Redis pipeline that queues commands and runs them on execute.

    Attributes:
        redis_client (StandInRedis): The stand-in client the commands run against.
        queued (list): The queued commands, as (method, arguments) pairs.
    """

    def __init__(self, redis_client):
        """Builds an empty pipeline.

        Args:
            redis_client (StandInRedis): The stand-in client the commands run against.

        Returns:
            None: This method returns nothing.
        """
        self.redis_client = redis_client
        self.queued = []

    def set(self, key, value):
        """Queues a string write.

        Args:
            key (str): The key to write.
            value (str): The value to store.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.set, (key, value)))

    def hset(self, key, mapping):
        """Queues a hash write.

        Args:
            key (str): The hash to write into.
            mapping (dict): The fields to write.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.hset, (key, mapping)))

    def hmget(self, key, fields):
        """Queues a read of several hash fields.

        Args:
            key (str): The hash to read.
            fields (list): The fields to read.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.hmget, (key, fields)))

    def expire(self, key, seconds):
        """Queues an expiry.

        Args:
            key (str): The key to expire.
            seconds (int): How long the key may live.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.expire, (key, seconds)))

    def execute(self):
        """Runs every queued command in order.

        Returns:
            list: Each command's reply, in the order queued.
        """
        replies = []
        for method, arguments in self.queued:
            replies.append(method(*arguments))
        self.queued = []
        return replies


class StandInRedis:
    """A stand-in for a Redis client that keeps strings and hashes in dictionaries.

    Attributes:
        values (dict): Each key's value: a string, or a dict for a hash.
        expiries (dict): Each key's expiry in seconds, as last set.
    """

    def __init__(self):
        """Builds an empty stand-in.

        Returns:
            None: This method returns nothing.
        """
        self.values = {}
        self.expiries = {}

    def pipeline(self, transaction=True):
        """Starts a pipeline.

        Args:
            transaction (bool): Ignored; the stand-in always runs commands in order.

        Returns:
            StandInPipeline: A new pipeline.
        """
        return StandInPipeline(self)

    def get(self, key):
        """Reads a string.

        Args:
            key (str): The key to read.

        Returns:
            str | None: The value, or None when the key is absent.
        """
        return self.values.get(key)

    def set(self, key, value):
        """Writes a string.

        Args:
            key (str): The key to write.
            value (str): The value to store.

        Returns:
            bool: Always True.
        """
        self.values[key] = value
        return True

    def exists(self, key):
        """Counts whether a key exists.

        Args:
            key (str): The key to test.

        Returns:
            int: 1 when the key exists, otherwise 0.
        """
        if key in self.values:
            return 1
        return 0

    def expire(self, key, seconds):
        """Records a key's expiry.

        Args:
            key (str): The key to expire.
            seconds (int): How long the key may live.

        Returns:
            bool: Always True.
        """
        self.expiries[key] = seconds
        return True

    def hset(self, key, mapping):
        """Writes fields into a hash.

        Args:
            key (str): The hash to write into.
            mapping (dict): The fields to write.

        Returns:
            int: The number of fields written.
        """
        stored = self.values.setdefault(key, {})
        stored.update(mapping)
        return len(mapping)

    def hmget(self, key, fields):
        """Reads several fields of a hash.

        Args:
            key (str): The hash to read.
            fields (list): The fields to read.

        Returns:
            list: Each field's value, or None where the field is absent.
        """
        stored = self.values.get(key, {})
        replies = []
        for field in fields:
            replies.append(stored.get(field))
        return replies

    def hgetall(self, key):
        """Reads a whole hash.

        Args:
            key (str): The hash to read.

        Returns:
            dict: The hash's fields, empty when the key is absent.
        """
        return dict(self.values.get(key, {}))


class StandInResult:
    """A stand-in for a SQLAlchemy result holding rows already fetched.

    Attributes:
        rows (list): The rows, in order.
    """

    def __init__(self, rows):
        """Holds the rows.

        Args:
            rows (list): The rows, in order.

        Returns:
            None: This method returns nothing.
        """
        self.rows = rows

    def __iter__(self):
        """Iterates over the rows.

        Returns:
            iterator: An iterator over the rows.
        """
        return iter(self.rows)

    def all(self):
        """Returns every row.

        Returns:
            list: The rows.
        """
        return list(self.rows)

    def scalar(self):
        """Returns the first row's single value.

        Returns:
            object: The first row, which for a scalar answer is the value itself, or None when there is no row.
        """
        if not self.rows:
            return None
        return self.rows[0]


class StandInConnection:
    """A stand-in for a SQLAlchemy connection that asks its engine for each statement's rows.

    Attributes:
        engine (StandInEngine): The engine holding the answers.
    """

    def __init__(self, engine):
        """Builds the connection.

        Args:
            engine (StandInEngine): The engine holding the answers.

        Returns:
            None: This method returns nothing.
        """
        self.engine = engine

    def __enter__(self):
        """Enters a `with` block.

        Returns:
            StandInConnection: This connection.
        """
        return self

    def __exit__(self, exception_type, exception, traceback):
        """Leaves a `with` block without suppressing any exception.

        Args:
            exception_type (type | None): The exception's class, if one was raised.
            exception (BaseException | None): The exception, if one was raised.
            traceback (types.TracebackType | None): The traceback, if one was raised.

        Returns:
            bool: Always False.
        """
        return False

    def execute(self, statement, parameters=None):
        """Answers one statement from the engine's rows.

        Args:
            statement (sqlalchemy.sql.elements.TextClause): The statement.
            parameters (dict | None): The bound parameters.

        Returns:
            StandInResult: The rows for the statement.
        """
        return self.engine.answer(str(statement), parameters)


class StandInEngine:
    """A stand-in for a SQLAlchemy engine that answers statements from rows held in memory.

    Each statement's rows are narrowed the way its WHERE clause would narrow them: to the broker it names, and to the tokens or instrument ids it lists.

    Attributes:
        answers (list): Pairs of (text fragment, rows); a statement containing the fragment gets the rows.
        calls (list): Pairs of (text fragment, parameters) for every statement answered.
    """

    def __init__(self):
        """Builds an engine with no answers.

        Returns:
            None: This method returns nothing.
        """
        self.answers = []
        self.calls = []

    def add_answer(self, fragment, rows):
        """Adds the rows a statement containing a fragment should return.

        Args:
            fragment (str): Text that appears in the statement and nowhere in the others.
            rows (list): The rows to return.

        Returns:
            None: This method returns nothing.
        """
        self.answers.append((fragment, rows))

    def connect(self):
        """Opens a connection.

        Returns:
            StandInConnection: A new connection.
        """
        return StandInConnection(self)

    def answer(self, statement_text, parameters):
        """Finds the rows for one statement and records the call.

        Args:
            statement_text (str): The statement's SQL.
            parameters (dict | None): The bound parameters.

        Returns:
            StandInResult: The rows for the statement.

        Raises:
            ValueError: If no fragment matches the statement.
        """
        for fragment, rows in self.answers:
            if fragment in statement_text:
                self.calls.append((fragment, parameters))
                kept = []
                for row in rows:
                    if self.keeps(row, parameters):
                        kept.append(row)
                return StandInResult(kept)
        raise ValueError(f'No stand-in answer for statement: {statement_text}')

    def keeps(self, row, parameters):
        """Says whether a row passes a statement's broker, token and instrument id filters.

        Args:
            row (types.SimpleNamespace): The row.
            parameters (dict | None): The bound parameters.

        Returns:
            bool: True when the row passes every filter the parameters carry.
        """
        if parameters is None:
            return True
        if 'broker' in parameters and row.broker != parameters['broker']:
            return False
        if 'tokens' in parameters and row.broker_token not in parameters['tokens']:
            return False
        if 'instrument_identifiers' in parameters and str(row.instrument_id) not in parameters['instrument_identifiers']:
            return False
        return True


class ResolvingForAnOrderExample:
    """Answers each of the cache's questions once and prints the answers and the counters.

    Attributes:
        mapping_date (datetime.date): The mapping date the stand-in Redis says is current.
        redis_client (StandInRedis): The Redis stand-in.
        engine (StandInEngine): The Postgres stand-in.
        cache (MappingCache): The cache being shown.
    """

    INFY_NSE_ID = '0a6e2d41-9c7f-4b58-8d13-6f2e4a9b7c05'
    BSE_1594_ID = '6c4a2e08-1f3d-4b5a-9c7e-0d2f4a6c8e13'

    def __init__(self):
        """Builds the stand-ins and the cache.

        Returns:
            None: This method returns nothing.
        """
        self.mapping_date = datetime.date(2026, 9, 30)
        self.redis_client = StandInRedis()
        self.redis_client.set('unified:catalogue:current_date', self.mapping_date.isoformat())
        self.engine = StandInEngine()
        self.engine.add_answer('b.broker_token = ANY', [
            self.candidate_row('dhan', '1594', self.BSE_1594_ID, 'bse', 'bse_equities', 'GANESHHOU'),
            self.candidate_row('dhan', '1594', self.INFY_NSE_ID, 'nse', 'nse_equities', 'INFY'),
        ])
        self.engine.add_answer('SELECT instrument_id, broker, broker_token', [
            self.handle_row(self.INFY_NSE_ID, 'dhan', '1594', None),
            self.handle_row(self.INFY_NSE_ID, 'kotak', '1594', 'INFY-EQ'),
            self.handle_row(self.INFY_NSE_ID, 'zerodha', '408065', 'INFY'),
        ])
        self.cache = MappingCache(self.engine, MappingRedisConnection(self.redis_client))

    def candidate_row(self, broker, broker_token, instrument_identifier, exchange, segment, symbol):
        """Builds one row of the token candidate join for an equity.

        Args:
            broker (str): The broker name.
            broker_token (str): The broker's token.
            instrument_identifier (str): The instrument id.
            exchange (str): The exchange.
            segment (str): The exchange-prefixed segment.
            symbol (str): The symbol.

        Returns:
            types.SimpleNamespace: The row.
        """
        return types.SimpleNamespace(
            broker=broker,
            broker_token=broker_token,
            mapping_date=self.mapping_date,
            instrument_id=uuid.UUID(instrument_identifier),
            exchange=exchange,
            segment=segment,
            shape='security',
            symbol=symbol,
            underlying_symbol=None,
            expiry_date=None,
            strike_price=None,
            option_type=None,
        )

    def handle_row(self, instrument_identifier, broker, broker_token, order_symbol):
        """Builds one row of `unified.broker_mappings` carrying an equity's order handle.

        Args:
            instrument_identifier (str): The instrument id.
            broker (str): The broker name.
            broker_token (str): The broker's token.
            order_symbol (str | None): The symbol the broker orders by, if any.

        Returns:
            types.SimpleNamespace: The row.
        """
        return types.SimpleNamespace(
            instrument_id=uuid.UUID(instrument_identifier),
            broker=broker,
            broker_token=broker_token,
            order_symbol=order_symbol,
            lot_size=decimal.Decimal('1'),
            tick_size=decimal.Decimal('0.10'),
        )

    def run(self):
        """Asks each question and prints the answer, then the counters and the queries.

        Returns:
            None: This method returns nothing.
        """
        print(f'The cache uses the engine it was given: {self.cache.engine is self.engine}')
        print(f'current_mapping_date: {self.cache.current_mapping_date()}')
        holiday = datetime.date(2026, 10, 2)
        print(f'resolves_to_current_date for {holiday}: {self.cache.resolves_to_current_date(holiday)}')
        tokens = [
            1594,
        ]
        unfiltered = self.cache.identities_for_tokens('dhan', tokens, holiday)
        print(f'identities_for_tokens, any segment: {unfiltered["1594"]["segment"]} {unfiltered["1594"]["symbol"]}')
        segments = [
            'nse_equities',
        ]
        filtered = self.cache.identities_for_tokens('dhan', tokens, self.mapping_date, segments)
        print(f'identities_for_tokens, nse_equities only: {filtered["1594"]["segment"]} {filtered["1594"]["symbol"]}')
        candidates = self.cache.candidates_for_tokens('dhan', tokens, self.mapping_date)
        for identity in candidates['1594']:
            print(f'candidates_for_tokens: {identity["segment"]} {identity["symbol"]}')
        instruments = [
            self.INFY_NSE_ID,
        ]
        identities = self.cache.identities_for_instruments(instruments, self.mapping_date)
        print(f'identities_for_instruments: {identities[self.INFY_NSE_ID]["symbol"]}')
        brokers = [
            'zerodha',
            'kotak',
        ]
        handles = self.cache.order_handles_for_instruments(instruments, self.mapping_date, brokers)
        print(f'order_handles_for_instruments, zerodha and kotak: {handles[self.INFY_NSE_ID]}')
        print(f'tokens_for_instruments, every broker: {self.cache.tokens_for_instruments(instruments, self.mapping_date)}')
        print(f'cache_statistics: {self.cache.cache_statistics()}')
        print('Postgres queries:')
        for fragment, parameters in self.engine.calls:
            print(f'  {fragment}')


if __name__ == '__main__':
    ResolvingForAnOrderExample().run()

It prints:

The cache uses the engine it was given: True
current_mapping_date: 2026-09-30
resolves_to_current_date for 2026-10-02: 2026-09-30
identities_for_tokens, any segment: bse_equities GANESHHOU
identities_for_tokens, nse_equities only: nse_equities INFY
candidates_for_tokens: bse_equities GANESHHOU
candidates_for_tokens: nse_equities INFY
identities_for_instruments: INFY
order_handles_for_instruments, zerodha and kotak: {'zerodha': {'broker_token': '408065', 'order_symbol': 'INFY', 'lot_size': '1', 'tick_size': '0.10'}, 'kotak': {'broker_token': '1594', 'order_symbol': 'INFY-EQ', 'lot_size': '1', 'tick_size': '0.10'}}
tokens_for_instruments, every broker: {'0a6e2d41-9c7f-4b58-8d13-6f2e4a9b7c05': {'dhan': '1594', 'kotak': '1594', 'zerodha': '408065'}}
cache_statistics: {'process_hits': 4, 'redis_hits': 0, 'postgres_hits': 2, 'misses': 0}
Postgres queries:
  b.broker_token = ANY
  SELECT instrument_id, broker, broker_token
examples/stock_brokers/instruments/mapping/utilities/cache/MappingCache/example_2_past_dates_warming_and_a_new_day.py
"""Shows the cache refusing a past date, warming a process up front, and moving to a new mapping date after a reset.

A `MappingCache` covers only the current mapping date. A question about an earlier date, such as a backtest asking about last week, is not a hot path, so `resolves_to_current_date` answers None for it and every look-up method raises `ValueError`; the caller is expected to test first and send such questions to `resolution.py`, which queries Postgres directly.

This program starts with a Redis that has never been warmed, so `current_mapping_date` falls back to reading the latest date from Postgres. It then calls `warm`, which a process that knows its universe can use at startup: it loads the order handles of the instruments named and every token of the brokers named, so that the first real look-up is already a process hit. After that `reset` drops everything this process holds, including the counters and the remembered date, and a new date published to Redis, as the next morning's warm would do, becomes the date the cache covers.

Two stand-ins replace the data stores, passed in through the cache's `engine` and `redis_connection` arguments. The Redis stand-in keeps strings and hashes in dictionaries. The Postgres engine stand-in holds the rows of each statement the cache runs, keeps only the broker, tokens and instrument ids a statement asks for, and records every call. Notice the counters before and after the warm, and that the look-ups after it reach neither Redis nor Postgres.

Run it from the project root:

    python examples/stock_brokers/instruments/mapping/utilities/cache/MappingCache/example_2_past_dates_warming_and_a_new_day.py
"""

import datetime
import decimal
import types
import uuid

from stock_brokers.instruments.mapping.utilities.cache import (
    MappingCache,
    MappingRedisConnection,
)


class StandInPipeline:
    """A stand-in for a Redis pipeline that queues commands and runs them on execute.

    Attributes:
        redis_client (StandInRedis): The stand-in client the commands run against.
        queued (list): The queued commands, as (method, arguments) pairs.
    """

    def __init__(self, redis_client):
        """Builds an empty pipeline.

        Args:
            redis_client (StandInRedis): The stand-in client the commands run against.

        Returns:
            None: This method returns nothing.
        """
        self.redis_client = redis_client
        self.queued = []

    def set(self, key, value):
        """Queues a string write.

        Args:
            key (str): The key to write.
            value (str): The value to store.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.set, (key, value)))

    def hset(self, key, mapping):
        """Queues a hash write.

        Args:
            key (str): The hash to write into.
            mapping (dict): The fields to write.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.hset, (key, mapping)))

    def hmget(self, key, fields):
        """Queues a read of several hash fields.

        Args:
            key (str): The hash to read.
            fields (list): The fields to read.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.hmget, (key, fields)))

    def expire(self, key, seconds):
        """Queues an expiry.

        Args:
            key (str): The key to expire.
            seconds (int): How long the key may live.

        Returns:
            None: This method returns nothing.
        """
        self.queued.append((self.redis_client.expire, (key, seconds)))

    def execute(self):
        """Runs every queued command in order.

        Returns:
            list: Each command's reply, in the order queued.
        """
        replies = []
        for method, arguments in self.queued:
            replies.append(method(*arguments))
        self.queued = []
        return replies


class StandInRedis:
    """A stand-in for a Redis client that keeps strings and hashes in dictionaries.

    Attributes:
        values (dict): Each key's value: a string, or a dict for a hash.
        expiries (dict): Each key's expiry in seconds, as last set.
    """

    def __init__(self):
        """Builds an empty stand-in.

        Returns:
            None: This method returns nothing.
        """
        self.values = {}
        self.expiries = {}

    def pipeline(self, transaction=True):
        """Starts a pipeline.

        Args:
            transaction (bool): Ignored; the stand-in always runs commands in order.

        Returns:
            StandInPipeline: A new pipeline.
        """
        return StandInPipeline(self)

    def get(self, key):
        """Reads a string.

        Args:
            key (str): The key to read.

        Returns:
            str | None: The value, or None when the key is absent.
        """
        return self.values.get(key)

    def set(self, key, value):
        """Writes a string.

        Args:
            key (str): The key to write.
            value (str): The value to store.

        Returns:
            bool: Always True.
        """
        self.values[key] = value
        return True

    def exists(self, key):
        """Counts whether a key exists.

        Args:
            key (str): The key to test.

        Returns:
            int: 1 when the key exists, otherwise 0.
        """
        if key in self.values:
            return 1
        return 0

    def expire(self, key, seconds):
        """Records a key's expiry.

        Args:
            key (str): The key to expire.
            seconds (int): How long the key may live.

        Returns:
            bool: Always True.
        """
        self.expiries[key] = seconds
        return True

    def hset(self, key, mapping):
        """Writes fields into a hash.

        Args:
            key (str): The hash to write into.
            mapping (dict): The fields to write.

        Returns:
            int: The number of fields written.
        """
        stored = self.values.setdefault(key, {})
        stored.update(mapping)
        return len(mapping)

    def hmget(self, key, fields):
        """Reads several fields of a hash.

        Args:
            key (str): The hash to read.
            fields (list): The fields to read.

        Returns:
            list: Each field's value, or None where the field is absent.
        """
        stored = self.values.get(key, {})
        replies = []
        for field in fields:
            replies.append(stored.get(field))
        return replies

    def hgetall(self, key):
        """Reads a whole hash.

        Args:
            key (str): The hash to read.

        Returns:
            dict: The hash's fields, empty when the key is absent.
        """
        return dict(self.values.get(key, {}))


class StandInResult:
    """A stand-in for a SQLAlchemy result holding rows already fetched.

    Attributes:
        rows (list): The rows, in order.
    """

    def __init__(self, rows):
        """Holds the rows.

        Args:
            rows (list): The rows, in order.

        Returns:
            None: This method returns nothing.
        """
        self.rows = rows

    def __iter__(self):
        """Iterates over the rows.

        Returns:
            iterator: An iterator over the rows.
        """
        return iter(self.rows)

    def all(self):
        """Returns every row.

        Returns:
            list: The rows.
        """
        return list(self.rows)

    def scalar(self):
        """Returns the first row's single value.

        Returns:
            object: The first row, which for a scalar answer is the value itself, or None when there is no row.
        """
        if not self.rows:
            return None
        return self.rows[0]


class StandInConnection:
    """A stand-in for a SQLAlchemy connection that asks its engine for each statement's rows.

    Attributes:
        engine (StandInEngine): The engine holding the answers.
    """

    def __init__(self, engine):
        """Builds the connection.

        Args:
            engine (StandInEngine): The engine holding the answers.

        Returns:
            None: This method returns nothing.
        """
        self.engine = engine

    def __enter__(self):
        """Enters a `with` block.

        Returns:
            StandInConnection: This connection.
        """
        return self

    def __exit__(self, exception_type, exception, traceback):
        """Leaves a `with` block without suppressing any exception.

        Args:
            exception_type (type | None): The exception's class, if one was raised.
            exception (BaseException | None): The exception, if one was raised.
            traceback (types.TracebackType | None): The traceback, if one was raised.

        Returns:
            bool: Always False.
        """
        return False

    def execute(self, statement, parameters=None):
        """Answers one statement from the engine's rows.

        Args:
            statement (sqlalchemy.sql.elements.TextClause): The statement.
            parameters (dict | None): The bound parameters.

        Returns:
            StandInResult: The rows for the statement.
        """
        return self.engine.answer(str(statement), parameters)


class StandInEngine:
    """A stand-in for a SQLAlchemy engine that answers statements from rows held in memory.

    Each statement's rows are narrowed the way its WHERE clause would narrow them: to the broker it names, and to the tokens or instrument ids it lists.

    Attributes:
        answers (list): Pairs of (text fragment, rows); a statement containing the fragment gets the rows.
        calls (list): Pairs of (text fragment, parameters) for every statement answered.
    """

    def __init__(self):
        """Builds an engine with no answers.

        Returns:
            None: This method returns nothing.
        """
        self.answers = []
        self.calls = []

    def add_answer(self, fragment, rows):
        """Adds the rows a statement containing a fragment should return.

        Args:
            fragment (str): Text that appears in the statement and nowhere in the others.
            rows (list): The rows to return.

        Returns:
            None: This method returns nothing.
        """
        self.answers.append((fragment, rows))

    def connect(self):
        """Opens a connection.

        Returns:
            StandInConnection: A new connection.
        """
        return StandInConnection(self)

    def answer(self, statement_text, parameters):
        """Finds the rows for one statement and records the call.

        Args:
            statement_text (str): The statement's SQL.
            parameters (dict | None): The bound parameters.

        Returns:
            StandInResult: The rows for the statement.

        Raises:
            ValueError: If no fragment matches the statement.
        """
        for fragment, rows in self.answers:
            if fragment in statement_text:
                self.calls.append((fragment, parameters))
                kept = []
                for row in rows:
                    if self.keeps(row, parameters):
                        kept.append(row)
                return StandInResult(kept)
        raise ValueError(f'No stand-in answer for statement: {statement_text}')

    def keeps(self, row, parameters):
        """Says whether a row passes a statement's broker, token and instrument id filters.

        Args:
            row (types.SimpleNamespace): The row.
            parameters (dict | None): The bound parameters.

        Returns:
            bool: True when the row passes every filter the parameters carry.
        """
        if parameters is None:
            return True
        if 'broker' in parameters and row.broker != parameters['broker']:
            return False
        if 'tokens' in parameters and row.broker_token not in parameters['tokens']:
            return False
        if 'instrument_identifiers' in parameters and str(row.instrument_id) not in parameters['instrument_identifiers']:
            return False
        return True


class PastDatesWarmingAndANewDayExample:
    """Refuses a past date, warms the cache, then resets it and follows a new mapping date.

    Attributes:
        mapping_date (datetime.date): The latest mapping date Postgres holds.
        redis_client (StandInRedis): The Redis stand-in, never warmed at first.
        engine (StandInEngine): The Postgres stand-in.
        cache (MappingCache): The cache being shown.
    """

    INFY_NSE_ID = '0a6e2d41-9c7f-4b58-8d13-6f2e4a9b7c05'
    HDFCBANK_NSE_ID = '5b7d9f1b-3d5f-4b7d-9f1b-3d5f7b9d1f68'

    def __init__(self):
        """Builds the stand-ins and the cache.

        Returns:
            None: This method returns nothing.
        """
        self.mapping_date = datetime.date(2026, 9, 30)
        self.redis_client = StandInRedis()
        self.engine = StandInEngine()
        self.engine.add_answer('max(mapping_date)', [
            self.mapping_date,
        ])
        self.engine.add_answer('SELECT DISTINCT broker_token', [
            types.SimpleNamespace(broker='zerodha', broker_token='408065'),
            types.SimpleNamespace(broker='zerodha', broker_token='341249'),
        ])
        self.engine.add_answer('b.broker_token = ANY', [
            self.candidate_row('zerodha', '341249', self.HDFCBANK_NSE_ID, 'nse', 'nse_equities', 'HDFCBANK'),
            self.candidate_row('zerodha', '408065', self.INFY_NSE_ID, 'nse', 'nse_equities', 'INFY'),
        ])
        self.engine.add_answer('SELECT instrument_id, broker, broker_token', [
            self.handle_row(self.INFY_NSE_ID, 'fyers', 'NSE:INFY-EQ', 'NSE:INFY-EQ'),
            self.handle_row(self.INFY_NSE_ID, 'zerodha', '408065', 'INFY'),
        ])
        self.cache = MappingCache(self.engine, MappingRedisConnection(self.redis_client))

    def refuse_a_past_date(self):
        """Asks about a date before the mapping date and prints the refusal.

        Returns:
            None: This method returns nothing.
        """
        last_week = datetime.date(2026, 9, 23)
        print(f'resolves_to_current_date for {last_week}: {self.cache.resolves_to_current_date(last_week)}')
        tokens = [
            '408065',
        ]
        try:
            self.cache.identities_for_tokens('zerodha', tokens, last_week)
        except ValueError as error:
            print(f'identities_for_tokens for {last_week}: ValueError: {error}')

    def warm_up_front(self):
        """Warms the cache for one instrument and one broker, then looks both up.

        Returns:
            None: This method returns nothing.
        """
        instruments = [
            self.INFY_NSE_ID,
        ]
        brokers = [
            'zerodha',
        ]
        loaded = self.cache.warm(self.mapping_date, instruments, brokers)
        print(f'warm loaded: {loaded}')
        print(f'  statistics after the warm: {self.cache.cache_statistics()}')
        queries_before = len(self.engine.calls)
        tokens = [
            '341249',
            '408065',
        ]
        identities = self.cache.identities_for_tokens('zerodha', tokens, self.mapping_date)
        for token in tokens:
            print(f'Token {token}: {identities[token]["symbol"]}')
        print(f'Order handles for INFY: {self.cache.order_handles_for_instruments(instruments, self.mapping_date)[self.INFY_NSE_ID]}')
        print(f'  statistics after the look-ups: {self.cache.cache_statistics()}')
        print(f'  queries the look-ups sent: {len(self.engine.calls) - queries_before}')

    def move_to_a_new_day(self):
        """Resets the cache and publishes the next mapping date to Redis.

        Returns:
            None: This method returns nothing.
        """
        next_day = datetime.date(2026, 10, 1)
        self.redis_client.set('unified:catalogue:current_date', next_day.isoformat())
        print(f'Before reset, still remembered: {self.cache.current_mapping_date()}')
        self.cache.reset()
        print(f'statistics after reset: {self.cache.cache_statistics()}')
        print(f'After reset, read from Redis: {self.cache.current_mapping_date()}')
        tokens = [
            '408065',
        ]
        try:
            self.cache.candidates_for_tokens('zerodha', tokens, self.mapping_date)
        except ValueError as error:
            print(f'candidates_for_tokens for {self.mapping_date}: ValueError: {error}')

    def run(self):
        """Reads the date from Postgres, refuses a past date, warms, then moves to a new day.

        Returns:
            None: This method returns nothing.
        """
        print(f'current_mapping_date with Redis never warmed: {self.cache.current_mapping_date()}')
        self.refuse_a_past_date()
        self.warm_up_front()
        self.move_to_a_new_day()

    def candidate_row(self, broker, broker_token, instrument_identifier, exchange, segment, symbol):
        """Builds one row of the token candidate join for an equity.

        Args:
            broker (str): The broker name.
            broker_token (str): The broker's token.
            instrument_identifier (str): The instrument id.
            exchange (str): The exchange.
            segment (str): The exchange-prefixed segment.
            symbol (str): The symbol.

        Returns:
            types.SimpleNamespace: The row.
        """
        return types.SimpleNamespace(
            broker=broker,
            broker_token=broker_token,
            mapping_date=self.mapping_date,
            instrument_id=uuid.UUID(instrument_identifier),
            exchange=exchange,
            segment=segment,
            shape='security',
            symbol=symbol,
            underlying_symbol=None,
            expiry_date=None,
            strike_price=None,
            option_type=None,
        )

    def handle_row(self, instrument_identifier, broker, broker_token, order_symbol):
        """Builds one row of `unified.broker_mappings` carrying an equity's order handle.

        Args:
            instrument_identifier (str): The instrument id.
            broker (str): The broker name.
            broker_token (str): The broker's token.
            order_symbol (str | None): The symbol the broker orders by, if any.

        Returns:
            types.SimpleNamespace: The row.
        """
        return types.SimpleNamespace(
            instrument_id=uuid.UUID(instrument_identifier),
            broker=broker,
            broker_token=broker_token,
            order_symbol=order_symbol,
            lot_size=decimal.Decimal('1'),
            tick_size=decimal.Decimal('0.10'),
        )


if __name__ == '__main__':
    PastDatesWarmingAndANewDayExample().run()

It prints:

current_mapping_date with Redis never warmed: 2026-09-30
resolves_to_current_date for 2026-09-23: None
identities_for_tokens for 2026-09-23: ValueError: the cache covers only the current mapping date, not 2026-09-23
warm loaded: {'instruments': 1, 'tokens': 2}
  statistics after the warm: {'process_hits': 0, 'redis_hits': 0, 'postgres_hits': 3, 'misses': 0}
Token 341249: HDFCBANK
Token 408065: INFY
Order handles for INFY: {'fyers': {'broker_token': 'NSE:INFY-EQ', 'order_symbol': 'NSE:INFY-EQ', 'lot_size': '1', 'tick_size': '0.10'}, 'zerodha': {'broker_token': '408065', 'order_symbol': 'INFY', 'lot_size': '1', 'tick_size': '0.10'}}
  statistics after the look-ups: {'process_hits': 3, 'redis_hits': 0, 'postgres_hits': 3, 'misses': 0}
  queries the look-ups sent: 0
Before reset, still remembered: 2026-09-30
statistics after reset: {'process_hits': 0, 'redis_hits': 0, 'postgres_hits': 0, 'misses': 0}
After reset, read from Redis: 2026-10-01
candidates_for_tokens for 2026-09-30: ValueError: the cache covers only the current mapping date, not 2026-09-30

Methods:

Name Description
current_mapping_date

The mapping date the cache covers, read from Redis and remembered in this process.
resolves_to_current_date

Decide whether a request for one as-of date can be answered from the cache.
identities_for_tokens

Resolve one broker's tokens to unified identities, through the cache.
candidates_for_tokens

Every identity one broker's tokens map to, through the cache, rather than one chosen identity.
identities_for_instruments

Resolve unified instrument ids to their identities, through the cache.
order_handles_for_instruments

Resolve unified identities to every broker's order handle for them, through the cache.
tokens_for_instruments

Resolve unified identities to every broker's token for them, through the cache.
warm

Fill this process's memory up front, for a process that knows its universe before it starts.
cache_statistics

How many look-ups each tier has answered in this process.
reset

Drop everything this process has cached, including the counts and the remembered mapping date.
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
2058
2059
2060
2061
2062
2063
2064
2065
2066
2067
2068
2069
2070
2071
2072
2073
2074
2075
2076
2077
2078
2079
2080
2081
2082
2083
def __init__(self, engine=None, redis_connection=None):
    """
    Build the cache and its tiers.

    Args:
        engine (sqlalchemy.engine.Engine): A SQLAlchemy engine over TimescaleDB. None builds one from the environment.
        redis_connection (MappingRedisConnection | None): The Redis connection to use. None builds one.

    Returns:
        None: This function returns nothing.
    """
    self.postgres_tier = MappingPostgresTier(engine)
    self.engine = self.postgres_tier.engine
    self.redis_tier = MappingRedisTier(redis_connection)
    self.statistics = {
        "process_hits": 0,
        "redis_hits": 0,
        "postgres_hits": 0,
        "misses": 0,
    }

    self._identity_lookup = InstrumentIdentityLookup(self.redis_tier, self.postgres_tier, self.statistics)
    self._order_handle_lookup = OrderHandleLookup(self.redis_tier, self.postgres_tier, self.statistics)
    self._token_lookups = {}
    self._mapping_date = None
    self._mapping_date_read_at = 0.0

current_mapping_date 

current_mapping_date()

The mapping date the cache covers, read from Redis and remembered in this process.

The date is re-read every five minutes rather than once, because a process that streams ticks all day would otherwise serve yesterday's mappings after the morning's download. When the date has moved, this process's own dictionaries are dropped, since every entry in them belongs to the date that has just been superseded.

Returns:

Type Description
date | None

The mapping date the cache covers, or None when nothing has been mapped at all.

Example usage

From example_1_resolving_for_an_order.py
print(f'current_mapping_date: {self.cache.current_mapping_date()}')
From example_2_past_dates_warming_and_a_new_day.py
print(f'Before reset, still remembered: {self.cache.current_mapping_date()}')
From example_2_past_dates_warming_and_a_new_day.py
print(f'After reset, read from Redis: {self.cache.current_mapping_date()}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
2085
2086
2087
2088
2089
2090
2091
2092
2093
2094
2095
2096
2097
2098
2099
2100
2101
2102
2103
2104
2105
2106
2107
def current_mapping_date(self):
    """
    The mapping date the cache covers, read from Redis and remembered in this process.

    The date is re-read every five minutes rather than once, because a process that streams ticks all day would otherwise serve yesterday's mappings after the morning's download. When the date has moved, this process's own dictionaries are dropped, since every entry in them belongs to the date that has just been superseded.

    Returns:
        datetime.date | None: The mapping date the cache covers, or None when nothing has been mapped at all.
    """
    if self._mapping_date is not None and time.monotonic() - self._mapping_date_read_at < CURRENT_DATE_REFRESH_SECONDS:
        return self._mapping_date

    mapping_date = self.redis_tier.read_current_date()
    if mapping_date is None:
        mapping_date = self.postgres_tier.latest_mapping_date()
    if mapping_date is None:
        return None

    if self._mapping_date is not None and mapping_date != self._mapping_date:
        self.reset()
    self._mapping_date = mapping_date
    self._mapping_date_read_at = time.monotonic()
    return self._mapping_date

resolves_to_current_date 

resolves_to_current_date(as_of_date)

Decide whether a request for one as-of date can be answered from the cache.

Parameters:

Name Type Description Default
as_of_date date

The date the caller asked about.

 required

Returns:

Type Description
date | None

The current mapping date when the request lands on it, or None when the request is for an earlier date and must go to Postgres.

Example usage

From example_1_resolving_for_an_order.py
print(f'resolves_to_current_date for {holiday}: {self.cache.resolves_to_current_date(holiday)}')
From example_2_past_dates_warming_and_a_new_day.py
print(f'resolves_to_current_date for {last_week}: {self.cache.resolves_to_current_date(last_week)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
2109
2110
2111
2112
2113
2114
2115
2116
2117
2118
2119
2120
2121
2122
2123
2124
def resolves_to_current_date(self, as_of_date):
    """
    Decide whether a request for one as-of date can be answered from the cache.

    Args:
        as_of_date (datetime.date): The date the caller asked about.

    Returns:
        datetime.date | None: The current mapping date when the request lands on it, or None when the request is for an earlier date and must go to Postgres.
    """
    mapping_date = self.current_mapping_date()
    if mapping_date is None:
        return None
    if as_of_date < mapping_date:
        return None
    return mapping_date

identities_for_tokens 

identities_for_tokens(broker, broker_tokens, as_of_date, segments=None)

Resolve one broker's tokens to unified identities, through the cache.

A request for a date earlier than the cached mapping date is refused here and belongs in resolution.py, which goes straight to Postgres for it.

Parameters:

Name Type Description Default
broker str

The broker name, for example "zerodha".

 required
broker_tokens list[str]

The broker's own tokens to resolve.

 required
as_of_date date

The date the caller asked about, which must land on the current mapping date.

 required
segments list[str] | None

Canonical segment names to restrict the answer to, or None to accept any segment.

 None

Returns:

Type Description
dict

Mapping of broker token to an identity dict. Tokens with no mapping, and tokens whose only mappings fall outside the requested segments, are absent.

Raises:

Type Description
ValueError

If the request is for a date the cache does not cover, which the caller should have tested with resolves_to_current_date.

Example usage

From example_1_resolving_for_an_order.py
unfiltered = self.cache.identities_for_tokens('dhan', tokens, holiday)
From example_1_resolving_for_an_order.py
filtered = self.cache.identities_for_tokens('dhan', tokens, self.mapping_date, segments)
From example_2_past_dates_warming_and_a_new_day.py
identities = self.cache.identities_for_tokens('zerodha', tokens, self.mapping_date)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
2126
2127
2128
2129
2130
2131
2132
2133
2134
2135
2136
2137
2138
2139
2140
2141
2142
2143
2144
2145
2146
2147
2148
2149
2150
2151
2152
2153
2154
2155
2156
2157
2158
2159
def identities_for_tokens(self, broker, broker_tokens, as_of_date, segments=None):
    """
    Resolve one broker's tokens to unified identities, through the cache.

    A request for a date earlier than the cached mapping date is refused here and belongs in resolution.py, which goes straight to Postgres for it.

    Args:
        broker (str): The broker name, for example "zerodha".
        broker_tokens (list[str]): The broker's own tokens to resolve.
        as_of_date (datetime.date): The date the caller asked about, which must land on the current mapping date.
        segments (list[str] | None): Canonical segment names to restrict the answer to, or None to accept any segment.

    Returns:
        dict: Mapping of broker token to an identity dict. Tokens with no mapping, and tokens whose only mappings fall outside the requested segments, are absent.

    Raises:
        ValueError: If the request is for a date the cache does not cover, which the caller should have tested with resolves_to_current_date.
    """
    mapping_date = self._mapping_date_for(as_of_date)
    if not broker_tokens:
        return {}

    tokens = []
    for token in broker_tokens:
        tokens.append(str(token))

    candidates_by_token = self._token_lookup(broker).resolve(mapping_date, tokens)

    resolved = {}
    for token, candidates in candidates_by_token.items():
        chosen = self._choose_candidate(candidates, segments)
        if chosen is not None:
            resolved[token] = chosen
    return resolved

candidates_for_tokens 

candidates_for_tokens(broker, broker_tokens, as_of_date, segments=None)

Every identity one broker's tokens map to, through the cache, rather than one chosen identity.

identities_for_tokens answers with the first candidate inside the segment filter, which is right for a caller that has to act on some answer. A caller that must not guess - the unified tick service, which would otherwise write one instrument's prices under another's id - needs to see that a token is ambiguous, and this gives it every candidate so it can decide.

Parameters:

Name Type Description Default
broker str

The broker name, for example "zerodha".

 required
broker_tokens list[str]

The broker's own tokens to resolve.

 required
as_of_date date

The date the caller asked about, which must land on the current mapping date.

 required
segments list[str] | None

Canonical segment names to restrict the answer to, or None to accept any segment.

 None

Returns:

Type Description
dict

Mapping of broker token to a list of identity dicts in tie-break order. Tokens with no candidate inside the requested segments are absent.

Raises:

Type Description
ValueError

If the request is for a date the cache does not cover, which the caller should have tested with resolves_to_current_date.

Example usage

From example_1_resolving_for_an_order.py
candidates = self.cache.candidates_for_tokens('dhan', tokens, self.mapping_date)
From example_2_past_dates_warming_and_a_new_day.py
self.cache.candidates_for_tokens('zerodha', tokens, self.mapping_date)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
2161
2162
2163
2164
2165
2166
2167
2168
2169
2170
2171
2172
2173
2174
2175
2176
2177
2178
2179
2180
2181
2182
2183
2184
2185
2186
2187
2188
2189
2190
2191
2192
2193
2194
2195
2196
2197
def candidates_for_tokens(self, broker, broker_tokens, as_of_date, segments=None):
    """
    Every identity one broker's tokens map to, through the cache, rather than one chosen identity.

    identities_for_tokens answers with the first candidate inside the segment filter, which is right for a caller that has to act on some answer. A caller that must not guess - the unified tick service, which would otherwise write one instrument's prices under another's id - needs to see that a token is ambiguous, and this gives it every candidate so it can decide.

    Args:
        broker (str): The broker name, for example "zerodha".
        broker_tokens (list[str]): The broker's own tokens to resolve.
        as_of_date (datetime.date): The date the caller asked about, which must land on the current mapping date.
        segments (list[str] | None): Canonical segment names to restrict the answer to, or None to accept any segment.

    Returns:
        dict: Mapping of broker token to a list of identity dicts in tie-break order. Tokens with no candidate inside the requested segments are absent.

    Raises:
        ValueError: If the request is for a date the cache does not cover, which the caller should have tested with resolves_to_current_date.
    """
    mapping_date = self._mapping_date_for(as_of_date)
    if not broker_tokens:
        return {}

    tokens = []
    for token in broker_tokens:
        tokens.append(str(token))

    candidates_by_token = self._token_lookup(broker).resolve(mapping_date, tokens)

    resolved = {}
    for token, candidates in candidates_by_token.items():
        kept = []
        for candidate in candidates:
            if segments is None or candidate["segment"] in segments:
                kept.append(candidate)
        if kept:
            resolved[token] = kept
    return resolved

identities_for_instruments 

identities_for_instruments(instrument_identifiers, as_of_date)

Resolve unified instrument ids to their identities, through the cache.

The reverse direction of identities_for_tokens. It shares the identity store the token path already fills rather than keeping one of its own, so an instrument this process has already seen from either direction costs a dictionary look-up.

Parameters:

Name Type Description Default
instrument_identifiers list[str]

The instrument ids to resolve.

 required
as_of_date date

The date the caller asked about, which must land on the current mapping date.

 required

Returns:

Type Description
dict

Mapping of instrument id to its identity dict. Instruments the master does not hold are absent.

Raises:

Type Description
ValueError

If the request is for a date the cache does not cover, which the caller should have tested with resolves_to_current_date.

Example usage

From example_1_resolving_for_an_order.py
identities = self.cache.identities_for_instruments(instruments, self.mapping_date)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
2199
2200
2201
2202
2203
2204
2205
2206
2207
2208
2209
2210
2211
2212
2213
2214
2215
2216
2217
2218
2219
2220
2221
2222
2223
def identities_for_instruments(self, instrument_identifiers, as_of_date):
    """
    Resolve unified instrument ids to their identities, through the cache.

    The reverse direction of identities_for_tokens. It shares the identity store the token path already fills rather than keeping one of its own, so an instrument this process has already seen from either direction costs a dictionary look-up.

    Args:
        instrument_identifiers (list[str]): The instrument ids to resolve.
        as_of_date (datetime.date): The date the caller asked about, which must land on the current mapping date.

    Returns:
        dict: Mapping of instrument id to its identity dict. Instruments the master does not hold are absent.

    Raises:
        ValueError: If the request is for a date the cache does not cover, which the caller should have tested with resolves_to_current_date.
    """
    mapping_date = self._mapping_date_for(as_of_date)
    if not instrument_identifiers:
        return {}

    identifiers = []
    for instrument_identifier in instrument_identifiers:
        identifiers.append(str(instrument_identifier))

    return self._identity_lookup.resolve(mapping_date, identifiers)

order_handles_for_instruments 

order_handles_for_instruments(instrument_identifiers, as_of_date, brokers=None)

Resolve unified identities to every broker's order handle for them, through the cache.

Parameters:

Name Type Description Default
instrument_identifiers list[str]

The instrument ids to resolve.

 required
as_of_date date

The date the caller asked about, which must land on the current mapping date.

 required
brokers list[str] | None

Broker names to restrict the answer to, or None for every broker that maps the instrument.

 None

Returns:

Type Description
dict

Mapping of instrument id to a dict of broker name to that broker's handle, itself a dict with keys "broker_token", "order_symbol", "lot_size" and "tick_size", where the last three are strings or None. Instruments with no mapping, and instruments no requested broker maps, are absent.

Raises:

Type Description
ValueError

If the request is for a date the cache does not cover, which the caller should have tested with resolves_to_current_date.

Example usage

From example_1_resolving_for_an_order.py
handles = self.cache.order_handles_for_instruments(instruments, self.mapping_date, brokers)
From example_2_past_dates_warming_and_a_new_day.py
print(f'Order handles for INFY: {self.cache.order_handles_for_instruments(instruments, self.mapping_date)[self.INFY_NSE_ID]}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
2225
2226
2227
2228
2229
2230
2231
2232
2233
2234
2235
2236
2237
2238
2239
2240
2241
2242
2243
2244
2245
2246
2247
2248
2249
2250
2251
2252
2253
2254
2255
def order_handles_for_instruments(self, instrument_identifiers, as_of_date, brokers=None):
    """
    Resolve unified identities to every broker's order handle for them, through the cache.

    Args:
        instrument_identifiers (list[str]): The instrument ids to resolve.
        as_of_date (datetime.date): The date the caller asked about, which must land on the current mapping date.
        brokers (list[str] | None): Broker names to restrict the answer to, or None for every broker that maps the instrument.

    Returns:
        dict: Mapping of instrument id to a dict of broker name to that broker's handle, itself a dict with keys "broker_token", "order_symbol", "lot_size" and "tick_size", where the last three are strings or None. Instruments with no mapping, and instruments no requested broker maps, are absent.

    Raises:
        ValueError: If the request is for a date the cache does not cover, which the caller should have tested with resolves_to_current_date.
    """
    mapping_date = self._mapping_date_for(as_of_date)
    if not instrument_identifiers:
        return {}

    identifiers = []
    for instrument_identifier in instrument_identifiers:
        identifiers.append(str(instrument_identifier))

    handles_by_instrument = self._order_handle_lookup.resolve(mapping_date, identifiers)

    resolved = {}
    for instrument_identifier, handles_by_broker in handles_by_instrument.items():
        narrowed = self._restrict_to_brokers(handles_by_broker, brokers)
        if narrowed:
            resolved[instrument_identifier] = narrowed
    return resolved

tokens_for_instruments 

tokens_for_instruments(instrument_identifiers, as_of_date, brokers=None)

Resolve unified identities to every broker's token for them, through the cache.

A projection of order_handles_for_instruments, for a caller that wants only the token. It shares that method's entries rather than keeping its own, so a market feed subscribing to an instrument and an order placed on the same instrument cost one query between them rather than two.

Parameters:

Name Type Description Default
instrument_identifiers list[str]

The instrument ids to resolve.

 required
as_of_date date

The date the caller asked about, which must land on the current mapping date.

 required
brokers list[str] | None

Broker names to restrict the answer to, or None for every broker that maps the instrument.

 None

Returns:

Type Description
dict

Mapping of instrument id to a dict of broker name to broker token. Instruments with no mapping, and instruments no requested broker maps, are absent.

Raises:

Type Description
ValueError

If the request is for a date the cache does not cover, which the caller should have tested with resolves_to_current_date.

Example usage

From example_1_resolving_for_an_order.py
print(f'tokens_for_instruments, every broker: {self.cache.tokens_for_instruments(instruments, self.mapping_date)}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
2257
2258
2259
2260
2261
2262
2263
2264
2265
2266
2267
2268
2269
2270
2271
2272
2273
2274
2275
2276
2277
2278
2279
def tokens_for_instruments(self, instrument_identifiers, as_of_date, brokers=None):
    """
    Resolve unified identities to every broker's token for them, through the cache.

    A projection of order_handles_for_instruments, for a caller that wants only the token. It shares that method's entries rather than keeping its own, so a market feed subscribing to an instrument and an order placed on the same instrument cost one query between them rather than two.

    Args:
        instrument_identifiers (list[str]): The instrument ids to resolve.
        as_of_date (datetime.date): The date the caller asked about, which must land on the current mapping date.
        brokers (list[str] | None): Broker names to restrict the answer to, or None for every broker that maps the instrument.

    Returns:
        dict: Mapping of instrument id to a dict of broker name to broker token. Instruments with no mapping, and instruments no requested broker maps, are absent.

    Raises:
        ValueError: If the request is for a date the cache does not cover, which the caller should have tested with resolves_to_current_date.
    """
    handles = self.order_handles_for_instruments(instrument_identifiers, as_of_date, brokers)
    tokens = {}
    for instrument_identifier, handles_by_broker in handles.items():
        for broker, handle in handles_by_broker.items():
            tokens.setdefault(instrument_identifier, {})[broker] = handle["broker_token"]
    return tokens

warm 

warm(as_of_date, instrument_identifiers=None, brokers=None)

Fill this process's memory up front, for a process that knows its universe before it starts.

This is deliberately optional and never happens by itself. Loading every broker's mappings into one process costs 4.06 seconds and 829 megabytes, which is why the default is to cache only what the process actually looks up. One broker alone costs about 0.21 seconds and 51 megabytes, which a strategy that knows its universe can reasonably pay at startup.

Parameters:

Name Type Description Default
as_of_date date

The date to warm for, which must land on the current mapping date.

 required
instrument_identifiers list[str] | None

Instruments whose broker tokens to load, or None to load none.

 None
brokers list[str] | None

Brokers whose entire token map to load, or None to load none.

 None

Returns:

Type Description
dict

The counts loaded, with keys "instruments" and "tokens".

Raises:

Type Description
ValueError

If the request is for a date the cache does not cover.

Example usage

From example_2_past_dates_warming_and_a_new_day.py
loaded = self.cache.warm(self.mapping_date, instruments, brokers)
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
2281
2282
2283
2284
2285
2286
2287
2288
2289
2290
2291
2292
2293
2294
2295
2296
2297
2298
2299
2300
2301
2302
2303
2304
2305
2306
2307
2308
2309
2310
2311
2312
2313
2314
2315
2316
def warm(self, as_of_date, instrument_identifiers=None, brokers=None):
    """
    Fill this process's memory up front, for a process that knows its universe before it starts.

    This is deliberately optional and never happens by itself. Loading every broker's mappings into one process costs 4.06 seconds and 829 megabytes, which is why the default is to cache only what the process actually looks up. One broker alone costs about 0.21 seconds and 51 megabytes, which a strategy that knows its universe can reasonably pay at startup.

    Args:
        as_of_date (datetime.date): The date to warm for, which must land on the current mapping date.
        instrument_identifiers (list[str] | None): Instruments whose broker tokens to load, or None to load none.
        brokers (list[str] | None): Brokers whose entire token map to load, or None to load none.

    Returns:
        dict: The counts loaded, with keys "instruments" and "tokens".

    Raises:
        ValueError: If the request is for a date the cache does not cover.
    """
    mapping_date = self._mapping_date_for(as_of_date)

    loaded = {
        "instruments": 0,
        "tokens": 0,
    }

    if instrument_identifiers:
        resolved = self.tokens_for_instruments(instrument_identifiers, as_of_date)
        loaded["instruments"] = len(resolved)

    if brokers:
        for broker in brokers:
            tokens = self.postgres_tier.broker_tokens_on_date(mapping_date, broker)
            for start in range(0, len(tokens), READ_BATCH_TOKENS):
                self.identities_for_tokens(broker, tokens[start:start + READ_BATCH_TOKENS], as_of_date)
            loaded["tokens"] += len(tokens)

    return loaded

cache_statistics 

cache_statistics()

How many look-ups each tier has answered in this process.

A cache that silently never hits looks exactly like one that works, so the counts are part of the interface rather than decoration. They count items looked up, not calls made, since one call resolves many tokens at once.

Returns:

Type Description
dict

Counts with keys "process_hits", "redis_hits", "postgres_hits" and "misses", where a miss is an item nothing maps at all.

Example usage

From example_1_resolving_for_an_order.py
print(f'cache_statistics: {self.cache.cache_statistics()}')
From example_2_past_dates_warming_and_a_new_day.py
print(f'  statistics after the warm: {self.cache.cache_statistics()}')
From example_2_past_dates_warming_and_a_new_day.py
print(f'  statistics after the look-ups: {self.cache.cache_statistics()}')
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
2318
2319
2320
2321
2322
2323
2324
2325
2326
2327
def cache_statistics(self):
    """
    How many look-ups each tier has answered in this process.

    A cache that silently never hits looks exactly like one that works, so the counts are part of the interface rather than decoration. They count items looked up, not calls made, since one call resolves many tokens at once.

    Returns:
        dict: Counts with keys "process_hits", "redis_hits", "postgres_hits" and "misses", where a miss is an item nothing maps at all.
    """
    return dict(self.statistics)

reset 

reset()

Drop everything this process has cached, including the counts and the remembered mapping date.

This exists for the verification report, which has to resolve a sample cold and then again warm, and for the day boundary, where every cached entry belongs to a mapping date that has just been superseded.

Returns:

Type Description
None

This function returns nothing.

Example usage

From example_2_past_dates_warming_and_a_new_day.py
self.cache.reset()
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
2329
2330
2331
2332
2333
2334
2335
2336
2337
2338
2339
2340
2341
2342
2343
2344
2345
def reset(self):
    """
    Drop everything this process has cached, including the counts and the remembered mapping date.

    This exists for the verification report, which has to resolve a sample cold and then again warm, and for the day boundary, where every cached entry belongs to a mapping date that has just been superseded.

    Returns:
        None: This function returns nothing.
    """
    self._identity_lookup.reset()
    self._order_handle_lookup.reset()
    for lookup in self._token_lookups.values():
        lookup.reset()
    for counter in self.statistics:
        self.statistics[counter] = 0
    self._mapping_date = None
    self._mapping_date_read_at = 0.0

_mapping_date_for 

_mapping_date_for(as_of_date)

The mapping date a request may be answered on, or a refusal.

Parameters:

Name Type Description Default
as_of_date date

The date the caller asked about.

 required

Returns:

Type Description
date

The current mapping date.

Raises:

Type Description
ValueError

If the request is for a date the cache does not cover.
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
2347
2348
2349
2350
2351
2352
2353
2354
2355
2356
2357
2358
2359
2360
2361
2362
2363
def _mapping_date_for(self, as_of_date):
    """
    The mapping date a request may be answered on, or a refusal.

    Args:
        as_of_date (datetime.date): The date the caller asked about.

    Returns:
        datetime.date: The current mapping date.

    Raises:
        ValueError: If the request is for a date the cache does not cover.
    """
    mapping_date = self.resolves_to_current_date(as_of_date)
    if mapping_date is None:
        raise ValueError(f"the cache covers only the current mapping date, not {as_of_date}")
    return mapping_date

_token_lookup 

_token_lookup(broker)

The token lookup for one broker, built on first use.

Parameters:

Name Type Description Default
broker str

The broker name, for example "zerodha".

 required

Returns:

Type Description
TokenCandidateLookup

That broker's lookup.
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
2365
2366
2367
2368
2369
2370
2371
2372
2373
2374
2375
2376
2377
2378
2379
2380
2381
2382
2383
2384
2385
def _token_lookup(self, broker):
    """
    The token lookup for one broker, built on first use.

    Args:
        broker (str): The broker name, for example "zerodha".

    Returns:
        TokenCandidateLookup: That broker's lookup.
    """
    lookup = self._token_lookups.get(broker)
    if lookup is None:
        lookup = TokenCandidateLookup(
            self.redis_tier,
            self.postgres_tier,
            self.statistics,
            broker,
            self._identity_lookup,
        )
        self._token_lookups[broker] = lookup
    return lookup

_choose_candidate 

_choose_candidate(candidates, segments)

Pick one identity out of a token's candidates, honouring an optional segment filter.

Parameters:

Name Type Description Default
candidates list[dict]

The token's identities, in tie-break order so the choice is stable between calls.

 required
segments list[str] | None

Canonical segment names to restrict the answer to, or None to accept any segment.

 required

Returns:

Type Description
dict | None

The chosen identity, or None when no candidate is in the requested segments.
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
2387
2388
2389
2390
2391
2392
2393
2394
2395
2396
2397
2398
2399
2400
2401
def _choose_candidate(self, candidates, segments):
    """
    Pick one identity out of a token's candidates, honouring an optional segment filter.

    Args:
        candidates (list[dict]): The token's identities, in tie-break order so the choice is stable between calls.
        segments (list[str] | None): Canonical segment names to restrict the answer to, or None to accept any segment.

    Returns:
        dict | None: The chosen identity, or None when no candidate is in the requested segments.
    """
    for candidate in candidates:
        if segments is None or candidate["segment"] in segments:
            return candidate
    return None

_restrict_to_brokers 

_restrict_to_brokers(handles_by_broker, brokers)

Narrow one instrument's order handles to the brokers the caller asked for.

Parameters:

Name Type Description Default
handles_by_broker dict

Mapping of broker name to that broker's handle.

 required
brokers list[str] | None

Broker names to keep, or None to keep every one.

 required

Returns:

Type Description
dict

The mapping, narrowed to the requested brokers.
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
2403
2404
2405
2406
2407
2408
2409
2410
2411
2412
2413
2414
2415
2416
2417
2418
2419
2420
def _restrict_to_brokers(self, handles_by_broker, brokers):
    """
    Narrow one instrument's order handles to the brokers the caller asked for.

    Args:
        handles_by_broker (dict): Mapping of broker name to that broker's handle.
        brokers (list[str] | None): Broker names to keep, or None to keep every one.

    Returns:
        dict: The mapping, narrowed to the requested brokers.
    """
    if brokers is None:
        return handles_by_broker
    narrowed = {}
    for broker in brokers:
        if broker in handles_by_broker:
            narrowed[broker] = handles_by_broker[broker]
    return narrowed

as_text 

as_text(value)

Decode one value Redis returned, which is bytes unless the client was built to decode responses.

Parameters:

Name Type Description Default
value bytes | str | None

The value as Redis returned it.

 required

Returns:

Type Description
str | None

The value as text, or None when there was none.
Source code in stock_brokers/instruments/mapping/utilities/cache.py 
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
def as_text(value):
    """
    Decode one value Redis returned, which is bytes unless the client was built to decode responses.

    Args:
        value (bytes | str | None): The value as Redis returned it.

    Returns:
        str | None: The value as text, or None when there was none.
    """
    if value is None:
        return None
    if isinstance(value, bytes):
        return value.decode()
    return value