factors ¶

Derive split, bonus and demerger factors from Yahoo Finance and the stored raw prices.

For each instrument stored unadjusted, the ratio q = Yahoo Close / raw close is taken on every day both have. Yahoo's Close is adjusted and the raw close is not, so q is 1 after the last corporate action and steps down at each action going back in time: RELIANCE's q is 1 from 2024-10-28, 0.5 before that bonus, and 0.4615 before the 2023-07-20 demerger of Jio Financial, whose factor of about 0.923 Yahoo builds into Close without ever listing it as an event.

Steps. q is walked from the newest day back. A move away from the current level counts as a step only when it holds for PERSISTENCE trading days, which passes over Yahoo's one-day bad prints (2025-03-18 across much of the market; NIFTYBEES for two days in December 2019). The tolerance widens for low prices, where Yahoo's rounding of the adjusted price to two decimals is a larger share of it. A step's size is the ratio of the median q on either side of it.

Classification. Each step is matched to a listed Yahoo event within MATCH_WINDOW trading days:

a step equal to the event's factor confirms a split at the exact ratio;
a step below it confirms the split and adds a provisional unclassified row for the remainder;
a step with no event is unclassified and provisional, with a note of whether the raw close fell by about the same amount. Demergers land here, but so do rights issues - Yahoo adjusts Close for those too - and inconsistencies in either series; across the first full build, 426 such steps met the old test for a demerger, most of them rights issues (UPL, SUZLON, PNBHOUSING) or whole ratios left over from a double adjustment (NAZARA's 0.5, RHETAN's 0.1). A person confirms a demerger.

Only listed split events are therefore confirmed automatically.

A listed event with no step anywhere inside the stored history is rejected, because applying it would be wrong for these stored prices whatever the reason. Usually the stored prices already include it: INDIAGLYCO's 1:1 bonus of 2025-08-12 was real, and its BSE series steps there, but flattrade's NSE series for it was served already halved before that date, so on NSE there is nothing left to apply. Events from before the stored history cannot be checked and are stored provisional.

A rebuild replaces the derived rows for an instrument: derived rows it no longer produces are deleted, so a step that moves - when a hole in the bars is filled - is not applied twice.

Raw gaps. An overnight move in the raw close beyond RAW_GAP with no factor near it is recorded as a provisional raw_gap row for a person to look at, with the evidence gathered: the ratio, the nearest simple fraction, whether the series changed token that day, and the factor dhan's re-adjusted history implies. This is how an action Yahoo has not caught up with surfaces - INDIAGLYCO on 2026-09-02.

Rows a person set (source = 'manual') are never touched, and rejected rows are never revived.

Classes:

Name	Description
`FactorBuilder`	Builds unified.adjustment_factors for the instruments stored unadjusted.

Functions:

Name	Description
`tolerance`	How far q may wander from its level without being a move, for a given price.
`find_steps`	The persistent level changes in q, newest first.
`exact_ratio`	A split ratio as shares after and before.

FactorBuilder ¶

FactorBuilder(yahoo=None)

Builds unified.adjustment_factors for the instruments stored unadjusted.

Attributes:

Name	Type	Description
`connection`		The psycopg2 connection.
`yahoo`	`YahooClient`	Paced Yahoo access.
`counts`	`dict`	What was derived, for the run summary.

Prepare a build.

Parameters:

Name	Type	Description	Default
`yahoo`	`YahooClient \| None`	A client to share, or None to build one.	`None`

Returns:

Type	Description
`None`	This function returns nothing.

Example programs

A bonus confirmed by yahooSteps yahoo does not explain

examples/stock_brokers/instruments/historical/utilities/unified/factors/FactorBuilder/example_1_a_bonus_confirmed_by_yahoo.py

"""Builds RELIANCE's adjustment factor for its 1:1 bonus of 2024-10-28, confirmed against the event Yahoo lists.

`FactorBuilder` derives split, bonus and demerger factors for every instrument stored unadjusted. For each one it reads the raw daily closes, fetches Yahoo's history, and takes the ratio q of Yahoo's close to the raw close on every day both have. Yahoo's close is adjusted and the raw close is not, so q is 1 after the last corporate action and steps down at each one going back in time. A step that matches an event Yahoo lists is a confirmed split at the exact ratio; here q is 0.5 before the bonus and 1 from it, and Yahoo lists the bonus as a ratio of 2.0, so the builder stores a confirmed 2-for-1 factor that halves earlier prices and doubles earlier volumes. The 50% overnight fall in the raw close is explained by that factor, so it is not reported as a raw gap.

The builder's constructor opens a PostgreSQL connection through `get_postgres`, and the example runner points PostgreSQL at a closed port, so the program replaces `get_postgres` in the factors module with a stand-in that returns an in-memory connection. That connection answers the builder's queries with 34 recorded-looking raw closes for October and November 2024 and records every row written. The builder accepts its Yahoo client as an argument, so a stand-in client is passed in; it answers with closes equal to the raw ones times q, rounded to the paisa as Yahoo rounds them, and never touches the network. The program first calls `instruments` and `raw_daily` to show what the build reads, then runs `build`.

Run it from the project root:

    python examples/stock_brokers/instruments/historical/utilities/unified/factors/FactorBuilder/example_1_a_bonus_confirmed_by_yahoo.py
"""

import datetime

from stock_brokers.instruments.historical.utilities.unified import factors
from stock_brokers.instruments.historical.utilities.unified.factors import (
    FactorBuilder,
)

BONUS_DATE = datetime.date(2024, 10, 28)


class RawHistory:
    """RELIANCE's raw daily closes around its bonus, as the unified price table would hold them.

    Attributes:
        days (list): The trading dates, oldest first.
        closes (list): The raw close on each date.
    """

    def __init__(self):
        """Builds the weekdays from 2024-10-01 to 2024-11-15 with closes near 2,700 before the bonus and 1,350 from it.

        Returns:
            None: This method returns nothing.
        """
        self.days = []
        self.closes = []
        day = datetime.date(2024, 10, 1)
        position = 0
        while day <= datetime.date(2024, 11, 15):
            if day.weekday() < 5:
                wiggle = (position % 4) * 6.5
                if day < BONUS_DATE:
                    self.closes.append(2700.0 + wiggle)
                else:
                    self.closes.append(1350.0 + wiggle / 2)
                self.days.append(day)
                position += 1
            day += datetime.timedelta(days=1)


class StandInYahooClient:
    """A stand-in for `YahooClient` that answers with closes adjusted for the bonus.

    Attributes:
        raw (RawHistory): The raw history the adjusted closes are made from.
        tickers (list): Every ticker asked for.
    """

    def __init__(self, raw):
        """Remembers the raw history.

        Args:
            raw (RawHistory): The raw history the adjusted closes are made from.

        Returns:
            None: This method returns nothing.
        """
        self.raw = raw
        self.tickers = []

    def history(self, ticker):
        """Answers Yahoo's closes and listed events for a ticker.

        Args:
            ticker (str): The Yahoo ticker.

        Returns:
            tuple: (closes by date, events by date).
        """
        self.tickers.append(ticker)
        closes = {}
        for day, raw_close in zip(self.raw.days, self.raw.closes):
            if day < BONUS_DATE:
                closes[day] = round(raw_close * 0.5, 2)
            else:
                closes[day] = raw_close
        events = {
            BONUS_DATE: 2.0,
        }
        return closes, events


class StandInCursor:
    """A stand-in for a psycopg2 cursor that hands every statement to its connection.

    Attributes:
        connection (StandInConnection): The connection that answers the statements.
        result (list): The rows the last statement produced.
        rowcount (int): How many rows the last statement changed.
    """

    def __init__(self, connection):
        """Builds a cursor over the stand-in connection.

        Args:
            connection (StandInConnection): The connection that answers the statements.

        Returns:
            None: This method returns nothing.
        """
        self.connection = connection
        self.result = []
        self.rowcount = 0

    def __enter__(self):
        """Opens the cursor as a context manager.

        Returns:
            StandInCursor: This cursor.
        """
        return self

    def __exit__(self, exception_type, exception, traceback):
        """Closes the cursor, letting any exception carry on.

        Args:
            exception_type (type | None): The class of an exception raised inside the block.
            exception (BaseException | None): The exception raised inside the block.
            traceback (types.TracebackType | None): Where it was raised.

        Returns:
            bool: Always False, so an exception is not swallowed.
        """
        return False

    def execute(self, statement, parameters):
        """Runs one statement against the stand-in connection.

        Args:
            statement (str): The SQL text.
            parameters (tuple | dict): The statement's parameters.

        Returns:
            None: This method returns nothing.
        """
        self.result, self.rowcount = self.connection.answer(statement, parameters)

    def fetchone(self):
        """The first row the last statement produced.

        Returns:
            tuple | None: The row, or None when there was none.
        """
        if not self.result:
            return None
        return self.result[0]

    def fetchall(self):
        """Every row the last statement produced.

        Returns:
            list: The rows.
        """
        return self.result


class StandInConnection:
    """A stand-in for a psycopg2 connection over RELIANCE's raw history.

    Attributes:
        raw (RawHistory): The raw history the price table query returns.
        factors_written (list): Every adjustment factor row inserted.
        fetches_written (list): Every Yahoo fetch state row written.
        commits (int): How many times the builder committed.
    """

    def __init__(self, raw):
        """Starts with nothing written.

        Args:
            raw (RawHistory): The raw history the price table query returns.

        Returns:
            None: This method returns nothing.
        """
        self.raw = raw
        self.factors_written = []
        self.fetches_written = []
        self.commits = 0

    def cursor(self):
        """Opens a stand-in cursor.

        Returns:
            StandInCursor: The cursor.
        """
        return StandInCursor(self)

    def answer(self, statement, parameters):
        """Answers one of the builder's statements.

        Args:
            statement (str): The SQL text.
            parameters (tuple | dict): The statement's parameters.

        Returns:
            tuple: (rows, rows changed).
        """
        if 'm.symbol not like' in statement:
            row = (
                'RELIANCE-NSE',
                'nse',
                'RELIANCE',
                None,
                False,
            )
            return [row], 0
        if 'close, source_id' in statement:
            rows = []
            for day, close in zip(self.raw.days, self.raw.closes):
                row = (
                    day,
                    close,
                    501,
                )
                rows.append(row)
            return rows, 0
        if "source = 'manual'" in statement and 'select ex_date' in statement:
            return [], 0
        if 'delete from' in statement:
            return [], 0
        if 'insert into unified.adjustment_factors' in statement:
            self.factors_written.append(parameters)
            return [], 1
        if 'insert into unified.yahoo_fetch_state' in statement:
            self.fetches_written.append(parameters)
            return [], 1
        raise ValueError(f'The stand-in does not know this statement: {statement[:60]}')

    def commit(self):
        """Counts a commit.

        Returns:
            None: This method returns nothing.
        """
        self.commits += 1

    def rollback(self):
        """Ends a read transaction, which the builder does before each Yahoo request.

        Returns:
            None: This method returns nothing.
        """


class StandInPostgres:
    """Hands out the stand-in connection in the place of `get_postgres`.

    Attributes:
        connection (StandInConnection): The connection handed out.
    """

    def __init__(self, connection):
        """Remembers the connection to hand out.

        Args:
            connection (StandInConnection): The connection handed out.

        Returns:
            None: This method returns nothing.
        """
        self.connection = connection

    def get_postgres(self):
        """Returns the stand-in connection.

        Returns:
            StandInConnection: The connection.
        """
        return self.connection


class BonusConfirmedByYahooExample:
    """Runs the factor build for RELIANCE and prints what it stored.

    Attributes:
        connection (StandInConnection): The in-memory connection.
        yahoo (StandInYahooClient): The stand-in Yahoo client.
        builder (FactorBuilder): The builder being shown.
    """

    def __init__(self):
        """Installs the stand-in connection and builds the builder with the stand-in Yahoo client.

        Returns:
            None: This method returns nothing.
        """
        raw = RawHistory()
        self.connection = StandInConnection(raw)
        factors.get_postgres = StandInPostgres(self.connection).get_postgres
        self.yahoo = StandInYahooClient(raw)
        self.builder = FactorBuilder(self.yahoo)

    def run(self):
        """Prints what the build reads, then builds and prints the stored rows.

        Returns:
            None: This method returns nothing.
        """
        print(f'Instruments to build: {self.builder.instruments()}')
        days, closes, source_ids = self.builder.raw_daily('RELIANCE-NSE')
        print(f'Raw closes: {len(days)} days from {days[0]} to {days[-1]}, {closes[0]} on the first, {closes[-1]} on the last, source {source_ids[0]}')
        counts = self.builder.build()
        print(f'Build summary: {counts}')
        print(f'Yahoo tickers asked for: {self.yahoo.tickers}')
        for row in self.connection.factors_written:
            print(f"Stored: {row['ex_date']} {row['kind']} {row['shares_after']}:{row['shares_before']} price factor {row['price_factor']} volume factor {row['volume_factor']} {row['status']} from {row['source']}")
            print(f"  evidence {row['evidence']}")
        for fetch in self.connection.fetches_written:
            print(f'Fetch state: ticker {fetch[1]} ({fetch[2]}), status {fetch[3]}, Yahoo history {fetch[5]} to {fetch[6]}, last event {fetch[7]}')
        print(f'Commits: {self.connection.commits}')


if __name__ == '__main__':
    BonusConfirmedByYahooExample().run()

It prints:

Instruments to build: [('RELIANCE-NSE', 'nse', 'RELIANCE', None, False)]
Raw closes: 34 days from 2024-10-01 to 2024-11-15, 2700.0 on the first, 1353.25 on the last, source 501
Build summary: {'stale_deleted': 0, 'split_confirmed': 1, 'ok': 1}
Yahoo tickers asked for: ['RELIANCE.NS']
Stored: 2024-10-28 split 2:1 price factor 0.5 volume factor 2.0 confirmed from yahoo_split_event
  evidence {"q_before": 0.5, "q_after": 1.0, "raw_close_before": 2713.0, "raw_close_after": 1359.75, "residual": 1.0}
Fetch state: ticker RELIANCE.NS (ticker), status ok, Yahoo history 2024-10-01 to 2024-11-15, last event 2024-10-28
Commits: 2

examples/stock_brokers/instruments/historical/utilities/unified/factors/FactorBuilder/example_2_steps_yahoo_does_not_explain.py

"""Shows what `FactorBuilder` does with evidence it cannot confirm by itself: an unlisted step, a listed event with no step, a raw gap, and a ticker that does not fit.

Only a step that matches a split Yahoo lists is confirmed automatically. This program calls the builder's steps one at a time to show the rest:

- `derive` on a demerger-shaped history: q steps from 0.923 to 1 on 2023-07-20 with no listed event, and the raw close falls by about the same amount, so the step is stored as a provisional `unclassified` row that notes the raw price fell alike, for a person to confirm. A split Yahoo lists on 2023-07-10, well inside the history but with no step in q, is `rejected`, because the stored prices already move with Yahoo's across it.
- `raw_gaps` on INDIAGLYCO's raw closes, which halve overnight on 2026-09-02 with nothing derived nearby: the move becomes a provisional `raw_gap` row carrying the evidence gathered, including that the series changed token that day and the factor Dhan's re-adjusted history implies, which `dhan_factor` reads.
- `store` writing those rows, except for a date a person has already decided by hand, and `record_fetch` noting a failed fetch.
- `build_one` for a BSE share whose every ticker form is a stub, which is recorded as `empty`, and for one whose newest q is 0.5 rather than 1, which means the ticker is not this instrument and is recorded as `mismatch`.

The constructor opens PostgreSQL through `get_postgres`, which the example runner points at a closed port, so the program replaces `get_postgres` in the factors module with a stand-in that returns an in-memory connection answering the builder's queries with canned rows and recording what is written. The Yahoo client is passed in as a stand-in, so the network is never touched.

Run it from the project root:

    python examples/stock_brokers/instruments/historical/utilities/unified/factors/FactorBuilder/example_2_steps_yahoo_does_not_explain.py
"""

import datetime

from stock_brokers.instruments.historical.utilities.unified import factors
from stock_brokers.instruments.historical.utilities.unified.factors import (
    FactorBuilder,
)

DEMERGER_DATE = datetime.date(2023, 7, 20)
GAP_DATE = datetime.date(2026, 9, 2)


class StandInYahooClient:
    """A stand-in for `YahooClient` that answers a stub for every BSE form and a mismatched history for one NSE ticker.

    Attributes:
        tickers (list): Every ticker asked for.
    """

    def __init__(self):
        """Starts with no tickers asked for.

        Returns:
            None: This method returns nothing.
        """
        self.tickers = []

    def history(self, ticker):
        """Answers Yahoo's closes and listed events for a ticker.

        Args:
            ticker (str): The Yahoo ticker.

        Returns:
            tuple: (closes by date, events by date).
        """
        self.tickers.append(ticker)
        closes = {}
        if ticker == 'WRONGCO.NS':
            day = datetime.date(2026, 9, 1)
            for position in range(12):
                closes[day + datetime.timedelta(days=position)] = 50.0
        if ticker == 'STUBCO.BO':
            closes[datetime.date(2026, 9, 1)] = 100.0
        return closes, {}


class StandInCursor:
    """A stand-in for a psycopg2 cursor that hands every statement to its connection.

    Attributes:
        connection (StandInConnection): The connection that answers the statements.
        result (list): The rows the last statement produced.
        rowcount (int): How many rows the last statement changed.
    """

    def __init__(self, connection):
        """Builds a cursor over the stand-in connection.

        Args:
            connection (StandInConnection): The connection that answers the statements.

        Returns:
            None: This method returns nothing.
        """
        self.connection = connection
        self.result = []
        self.rowcount = 0

    def __enter__(self):
        """Opens the cursor as a context manager.

        Returns:
            StandInCursor: This cursor.
        """
        return self

    def __exit__(self, exception_type, exception, traceback):
        """Closes the cursor, letting any exception carry on.

        Args:
            exception_type (type | None): The class of an exception raised inside the block.
            exception (BaseException | None): The exception raised inside the block.
            traceback (types.TracebackType | None): Where it was raised.

        Returns:
            bool: Always False, so an exception is not swallowed.
        """
        return False

    def execute(self, statement, parameters):
        """Runs one statement against the stand-in connection.

        Args:
            statement (str): The SQL text.
            parameters (tuple | dict): The statement's parameters.

        Returns:
            None: This method returns nothing.
        """
        self.result, self.rowcount = self.connection.answer(statement, parameters)

    def fetchone(self):
        """The first row the last statement produced.

        Returns:
            tuple | None: The row, or None when there was none.
        """
        if not self.result:
            return None
        return self.result[0]

    def fetchall(self):
        """Every row the last statement produced.

        Returns:
            list: The rows.
        """
        return self.result


class StandInConnection:
    """A stand-in for a psycopg2 connection with raw closes for two instruments and one Dhan bar.

    Attributes:
        raw (dict): Instrument id to its raw (date, close, source id) rows.
        factors_written (list): Every adjustment factor row inserted.
        fetches_written (list): Every Yahoo fetch state row written.
    """

    def __init__(self):
        """Fills in twelve days of raw closes for two instruments.

        Returns:
            None: This method returns nothing.
        """
        self.raw = {}
        self.factors_written = []
        self.fetches_written = []
        instrument_ids = [
            'STUBCO-BSE',
            'WRONGCO-NSE',
        ]
        for instrument_id in instrument_ids:
            rows = []
            day = datetime.date(2026, 9, 1)
            for position in range(12):
                row = (
                    day + datetime.timedelta(days=position),
                    100.0,
                    601,
                )
                rows.append(row)
            self.raw[instrument_id] = rows

    def cursor(self):
        """Opens a stand-in cursor.

        Returns:
            StandInCursor: The cursor.
        """
        return StandInCursor(self)

    def answer(self, statement, parameters):
        """Answers one of the builder's statements.

        Args:
            statement (str): The SQL text.
            parameters (tuple | dict): The statement's parameters.

        Returns:
            tuple: (rows, rows changed).
        """
        if 'close, source_id' in statement:
            return self.raw[parameters[0]], 0
        if 'join dhan.price_history' in statement:
            row = (
                242.5,
            )
            return [row], 0
        if "source = 'manual'" in statement and 'select ex_date' in statement:
            row = (
                datetime.date(2023, 7, 10),
            )
            return [row], 0
        if 'delete from' in statement:
            return [], 2
        if 'insert into unified.adjustment_factors' in statement:
            self.factors_written.append(parameters)
            return [], 1
        if 'insert into unified.yahoo_fetch_state' in statement:
            self.fetches_written.append(parameters)
            return [], 1
        raise ValueError(f'The stand-in does not know this statement: {statement[:60]}')

    def commit(self):
        """Accepts a commit.

        Returns:
            None: This method returns nothing.
        """

    def rollback(self):
        """Accepts a rollback.

        Returns:
            None: This method returns nothing.
        """


class StandInPostgres:
    """Hands out the stand-in connection in the place of `get_postgres`.

    Attributes:
        connection (StandInConnection): The connection handed out.
    """

    def __init__(self, connection):
        """Remembers the connection to hand out.

        Args:
            connection (StandInConnection): The connection handed out.

        Returns:
            None: This method returns nothing.
        """
        self.connection = connection

    def get_postgres(self):
        """Returns the stand-in connection.

        Returns:
            StandInConnection: The connection.
        """
        return self.connection


class StepsYahooDoesNotExplainExample:
    """Calls the builder's steps one at a time and prints what each decides.

    Attributes:
        connection (StandInConnection): The in-memory connection.
        yahoo (StandInYahooClient): The stand-in Yahoo client.
        builder (FactorBuilder): The builder being shown.
    """

    def __init__(self):
        """Installs the stand-in connection and builds the builder.

        Returns:
            None: This method returns nothing.
        """
        self.connection = StandInConnection()
        factors.get_postgres = StandInPostgres(self.connection).get_postgres
        self.yahoo = StandInYahooClient()
        self.builder = FactorBuilder(self.yahoo)

    def weekdays(self, first_day, count):
        """Lists a number of weekdays from a date.

        Args:
            first_day (datetime.date): The first date.
            count (int): How many weekdays to list.

        Returns:
            list: The dates.
        """
        days = []
        day = first_day
        while len(days) < count:
            if day.weekday() < 5:
                days.append(day)
            day += datetime.timedelta(days=1)
        return days

    def describe(self, row):
        """Prints one factor row on one line, with its evidence below.

        Args:
            row (dict): The row.

        Returns:
            None: This method returns nothing.
        """
        print(f"  {row['ex_date']} {row['kind']} {row['status']} from {row['source']}, price factor {round(row['price_factor'], 4)}")
        print(f"    evidence {row['evidence']}")

    def derive_demerger(self):
        """Derives rows for a demerger-shaped history with one listed event that shows no step.

        Returns:
            list: The derived rows.
        """
        days = self.weekdays(datetime.date(2023, 6, 26), 30)
        raw_closes = []
        ratios = []
        for day in days:
            if day < DEMERGER_DATE:
                raw_closes.append(2600.0)
                ratios.append(0.923)
            else:
                raw_closes.append(2400.0)
                ratios.append(1.0)
        events = {
            datetime.date(2023, 7, 10): 2.0,
        }
        return self.builder.derive('DEMERGE-NSE', 'DEMERGE.NS', days, raw_closes, ratios, events)

    def find_raw_gap(self):
        """Looks for overnight raw moves nothing explains in INDIAGLYCO's closes.

        Returns:
            list: The raw gap rows.
        """
        days = self.weekdays(datetime.date(2026, 8, 24), 12)
        raw_closes = []
        source_ids = []
        for day in days:
            if day < GAP_DATE:
                raw_closes.append(485.0)
                source_ids.append(701)
            else:
                raw_closes.append(242.5)
                source_ids.append(702)
        return self.builder.raw_gaps('INDIAGLYCO-NSE', 'nse', days, raw_closes, source_ids, [])

    def run(self):
        """Walks through each step and prints what it decided and wrote.

        Returns:
            None: This method returns nothing.
        """
        derived = self.derive_demerger()
        print('derive, for a demerger-shaped history:')
        for row in derived:
            self.describe(row)
        gaps = self.find_raw_gap()
        print('raw_gaps, for INDIAGLYCO:')
        for row in gaps:
            self.describe(row)
        dhan = self.builder.dhan_factor('INDIAGLYCO-NSE', 'nse', datetime.date(2026, 9, 1), 485.0)
        print(f'dhan_factor on 2026-09-01: {dhan}')
        self.builder.store('DEMERGE-NSE', derived)
        print(f'Rows stored: {len(self.connection.factors_written)}, the manually decided 2023-07-10 left alone')
        self.builder.record_fetch('INDIAGLYCO-NSE', None, None, 'error', 'Yahoo answered 404', None, None, None)
        self.builder.build_one('STUBCO-BSE', 'bse', 'STUBCO', '599999', False)
        self.builder.build_one('WRONGCO-NSE', 'nse', 'WRONGCO', None)
        print(f'Yahoo tickers asked for: {self.yahoo.tickers}')
        for fetch in self.connection.fetches_written:
            print(f'Fetch state for {fetch[0]}: ticker {fetch[1]}, status {fetch[3]}, error {fetch[4]}')
        self.builder.count('checked_by_hand')
        print(f'Counts: {self.builder.counts}')


if __name__ == '__main__':
    StepsYahooDoesNotExplainExample().run()

It prints:

derive, for a demerger-shaped history:
  2023-07-20 unclassified provisional from yahoo_close_ratio, price factor 0.923
    evidence {'q_before': 0.923, 'q_after': 1.0, 'raw_close_before': 2600.0, 'raw_close_after': 2400.0, 'raw_move': 0.923077, 'raw_price_fell_alike': True}
  2023-07-10 split rejected from yahoo_split_event, price factor 0.5
    evidence {'note': "listed by Yahoo, but Yahoo's Close and the stored prices move together across it: either there was no action, or the stored prices already include it"}
raw_gaps, for INDIAGLYCO:
  2026-09-02 unclassified provisional from raw_gap, price factor 0.5
    evidence {'raw_close_before': 485.0, 'raw_close_after': 242.5, 'move': 0.5, 'nearest_fraction': '1/2', 'token_changed': True, 'dhan_factor': 0.5}
dhan_factor on 2026-09-01: 0.5
Rows stored: 1, the manually decided 2023-07-10 left alone
Yahoo tickers asked for: ['STUBCO.BO', '599999.BO', 'WRONGCO.NS']
Fetch state for INDIAGLYCO-NSE: ticker None, status error, error Yahoo answered 404
Fetch state for STUBCO-BSE: ticker None, status empty, error no ticker form covers the raw history
Fetch state for WRONGCO-NSE: ticker WRONGCO.NS, status mismatch, error latest Yahoo/raw ratio 0.5000
Counts: {'stale_deleted': 2, 'unclassified_provisional': 1, 'empty': 1, 'mismatch': 1, 'checked_by_hand': 1}

Methods:

Name	Description
`count`	Add one to a summary count.
`instruments`	The instruments with unadjusted daily bars, and what is needed to find them on Yahoo.
`raw_daily`	One instrument's raw daily closes, and the source each came from.
`build`	Fetch, derive and store factors for each instrument.
`build_one`	Fetch one instrument from Yahoo, derive its factors and store them.
`derive`	Turn q's steps and Yahoo's events into factor rows.
`raw_gaps`	Overnight raw moves too large to be trading, with no factor near them.
`dhan_factor`	The factor dhan's adjusted history implies for a day: its close over the raw close.
`store`	Upsert derived rows, leaving manual and rejected rows as they are.
`record_fetch`	Record where the Yahoo fetch got to for an instrument.

Source code in stock_brokers/instruments/historical/utilities/unified/factors.py

def __init__(self, yahoo=None):
    """
    Prepare a build.

    Args:
        yahoo (YahooClient | None): A client to share, or None to build one.

    Returns:
        None: This function returns nothing.
    """
    self.connection = get_postgres()
    self.yahoo = yahoo or YahooClient()
    self.counts = {}

count ¶

count(key)

Add one to a summary count.

Example usage

From example_2_steps_yahoo_does_not_explain.py

self.builder.count('checked_by_hand')

Source code in stock_brokers/instruments/historical/utilities/unified/factors.py

def count(self, key):
    """Add one to a summary count."""
    self.counts[key] = self.counts.get(key, 0) + 1

instruments ¶

instruments(instrument_ids=None)

The instruments with unadjusted daily bars, and what is needed to find them on Yahoo.

Parameters:

Name	Type	Description	Default
`instrument_ids`	`set[str] \| None`	Restrict to these, or None for all.	`None`

Returns:

Type	Description
`list[tuple]`	(instrument_id, exchange, symbol, scrip_code, listed_on_nse), scrip_code None for NSE.

Example usage

From example_1_a_bonus_confirmed_by_yahoo.py

print(f'Instruments to build: {self.builder.instruments()}')

Source code in stock_brokers/instruments/historical/utilities/unified/factors.py

def instruments(self, instrument_ids=None):
    """
    The instruments with unadjusted daily bars, and what is needed to find them on Yahoo.

    Args:
        instrument_ids (set[str] | None): Restrict to these, or None for all.

    Returns:
        list[tuple]: (instrument_id, exchange, symbol, scrip_code, listed_on_nse), scrip_code None for NSE.
    """
    segments = [f"{exchange}_{segment}" for exchange in ("nse", "bse") for segment in ADJUSTABLE_SEGMENTS]
    with self.connection.cursor() as cursor:
        cursor.execute(f"""
            select m.instrument_id::text, m.exchange, m.symbol,
                   (select b.broker_token from {tables.BROKER_MAPPINGS} b
                    where b.instrument_id = m.instrument_id and b.broker = 'dhan' and m.exchange = 'bse'
                    order by b.mapping_date desc limit 1),
                   exists (select 1 from {tables.MASTER} n
                           where n.exchange = 'nse' and n.symbol = m.symbol
                             and n.segment = replace(m.segment, 'bse_', 'nse_'))
            from {tables.MASTER} m
            where m.segment = any(%s)
              and exists (select 1 from {tables.PRICE_HISTORY_SOURCES} s
                          where s.instrument_id = m.instrument_id and s."interval" = 'day'
                            and s.price_basis = 'unadjusted' and s.loaded_latest is not null)
              and m.symbol not like '%%NSETEST%%'
              and (%s::text[] is null or m.instrument_id::text = any(%s))
            order by m.exchange, m.symbol
        """, (segments, list(instrument_ids) if instrument_ids else None,
              list(instrument_ids) if instrument_ids else None))
        return cursor.fetchall()

raw_daily ¶

raw_daily(instrument_id)

One instrument's raw daily closes, and the source each came from.

Parameters:

Name	Type	Description	Default
`instrument_id`	`str`	The instrument.	required

Returns:

Type	Description
`tuple[list, list, list]`	Dates, closes and source ids, oldest first.

Example usage

From example_1_a_bonus_confirmed_by_yahoo.py

days, closes, source_ids = self.builder.raw_daily('RELIANCE-NSE')

Source code in stock_brokers/instruments/historical/utilities/unified/factors.py

def raw_daily(self, instrument_id):
    """
    One instrument's raw daily closes, and the source each came from.

    Args:
        instrument_id (str): The instrument.

    Returns:
        tuple[list, list, list]: Dates, closes and source ids, oldest first.
    """
    with self.connection.cursor() as cursor:
        cursor.execute(f"""
            select ("time" at time zone 'Asia/Kolkata')::date, close, source_id
            from {tables.PRICE_HISTORY}
            where instrument_id = %s and "interval" = 'day' and close > 0
            order by "time"
        """, (instrument_id,))
        rows = cursor.fetchall()
    # End the read transaction: the Yahoo request that follows can take seconds, and an open
    # transaction held across it keeps vacuum from cleaning the tables behind it.
    self.connection.rollback()
    return [row[0] for row in rows], [float(row[1]) for row in rows], [row[2] for row in rows]

build ¶

build(instrument_ids=None)

Fetch, derive and store factors for each instrument.

Parameters:

Name	Type	Description	Default
`instrument_ids`	`set[str] \| None`	Restrict to these, or None for all.	`None`

Returns:

Type	Description
`dict`	Summary counts.

Example usage

From example_1_a_bonus_confirmed_by_yahoo.py

counts = self.builder.build()

Source code in stock_brokers/instruments/historical/utilities/unified/factors.py

def build(self, instrument_ids=None):
    """
    Fetch, derive and store factors for each instrument.

    Args:
        instrument_ids (set[str] | None): Restrict to these, or None for all.

    Returns:
        dict: Summary counts.
    """
    targets = self.instruments(instrument_ids)
    LOGGER.info("factors: %d instruments", len(targets))
    for position, (instrument_id, exchange, symbol, scrip_code, listed_on_nse) in enumerate(targets, 1):
        try:
            self.build_one(instrument_id, exchange, symbol, scrip_code, listed_on_nse)
        except Exception as error:
            self.connection.rollback()
            self.record_fetch(instrument_id, None, None, "error", str(error)[:500], None, None, None)
            self.count("error")
            LOGGER.exception("factors: %s %s failed", exchange, symbol)
        if position % 100 == 0:
            LOGGER.info("factors: %d of %d, %s", position, len(targets), self.counts)
    return self.counts

build_one ¶

build_one(instrument_id, exchange, symbol, scrip_code, listed_on_nse=False)

Fetch one instrument from Yahoo, derive its factors and store them.

Parameters:

Name	Type	Description	Default
`instrument_id`	`str`	The instrument.	required
`exchange`	`str`	"nse" or "bse".	required
`symbol`	`str`	Its trading symbol.	required
`scrip_code`	`str \| None`	Its BSE scrip code.	required
`listed_on_nse`	`bool`	Whether a BSE instrument is also listed on NSE.	`False`

Returns:

Type	Description
`None`	This function returns nothing.

Example usage

From example_2_steps_yahoo_does_not_explain.py

self.builder.build_one('STUBCO-BSE', 'bse', 'STUBCO', '599999', False)

From example_2_steps_yahoo_does_not_explain.py

self.builder.build_one('WRONGCO-NSE', 'nse', 'WRONGCO', None)

Source code in stock_brokers/instruments/historical/utilities/unified/factors.py

def build_one(self, instrument_id, exchange, symbol, scrip_code, listed_on_nse=False):
    """
    Fetch one instrument from Yahoo, derive its factors and store them.

    Args:
        instrument_id (str): The instrument.
        exchange (str): "nse" or "bse".
        symbol (str): Its trading symbol.
        scrip_code (str | None): Its BSE scrip code.
        listed_on_nse (bool): Whether a BSE instrument is also listed on NSE.

    Returns:
        None: This function returns nothing.
    """
    days, raw_closes, source_ids = self.raw_daily(instrument_id)
    if len(days) < PERSISTENCE * 2:
        self.count("too_little_raw")
        return

    chosen = None
    for ticker, kind in ticker_forms(exchange, symbol, scrip_code, listed_on_nse):
        closes, events = self.yahoo.history(ticker)
        if coverage(closes, days) >= MINIMUM_COVERAGE:
            chosen = (ticker, kind, closes, events)
            break
    if chosen is None:
        self.record_fetch(instrument_id, None, None, "empty", "no ticker form covers the raw history",
                          None, None, None)
        self.count("empty")
        return
    ticker, kind, closes, events = chosen

    overlap = [index for index, day in enumerate(days) if day in closes]
    overlap_days = [days[index] for index in overlap]
    overlap_raw = [raw_closes[index] for index in overlap]
    ratios = [closes[day] / raw for day, raw in zip(overlap_days, overlap_raw)]
    history_start, history_end = min(closes), max(closes)
    last_event = max(events) if events else None

    # The newest day decides, not a median of the last few: an action that went ex yesterday leaves
    # one day at the current level, as PGIL's bonus of 2026-09-11 did.
    latest_level = ratios[-1]
    if abs(latest_level - 1) > LATEST_LEVEL_TOLERANCE:
        self.record_fetch(instrument_id, ticker, kind, "mismatch",
                          f"latest Yahoo/raw ratio {latest_level:.4f}", history_start, history_end, last_event)
        self.count("mismatch")
        return

    rows = self.derive(instrument_id, ticker, overlap_days, overlap_raw, ratios, events)
    rows.extend(self.raw_gaps(instrument_id, exchange, days, raw_closes, source_ids, rows))
    self.store(instrument_id, rows)
    self.record_fetch(instrument_id, ticker, kind, "ok", None, history_start, history_end, last_event)
    self.count("ok")

derive ¶

derive(instrument_id, ticker, days, raw_closes, ratios, events)

Turn q's steps and Yahoo's events into factor rows.

Parameters:

Name	Type	Description	Default
`instrument_id`	`str`	The instrument.	required
`ticker`	`str`	The Yahoo ticker used.	required
`days`	`list`	Overlap dates, oldest first.	required
`raw_closes`	`list[float]`	Raw closes on those dates.	required
`ratios`	`list[float]`	q on those dates.	required
`events`	`dict`	Yahoo's listed events, date to ratio.	required

Returns:

Type	Description
`list[dict]`	Factor rows.

Example usage

From example_2_steps_yahoo_does_not_explain.py

return self.builder.derive('DEMERGE-NSE', 'DEMERGE.NS', days, raw_closes, ratios, events)

Source code in stock_brokers/instruments/historical/utilities/unified/factors.py

def derive(self, instrument_id, ticker, days, raw_closes, ratios, events):
    """
    Turn q's steps and Yahoo's events into factor rows.

    Args:
        instrument_id (str): The instrument.
        ticker (str): The Yahoo ticker used.
        days (list): Overlap dates, oldest first.
        raw_closes (list[float]): Raw closes on those dates.
        ratios (list[float]): q on those dates.
        events (dict): Yahoo's listed events, date to ratio.

    Returns:
        list[dict]: Factor rows.
    """
    rows = []
    steps = find_steps(days, ratios, raw_closes)
    matched_events = set()
    for step in steps:
        index = step["index"]
        window = days[max(0, index - MATCH_WINDOW):min(len(days), index + MATCH_WINDOW + 1)]
        event_date = None
        for day in sorted(events, key=lambda day: abs((day - step["ex_date"]).days)):
            if window[0] <= day <= window[-1]:
                event_date = day
                break
        evidence = {"q_before": round(ratios[index - 1], 6), "q_after": round(ratios[index], 6),
                    "raw_close_before": raw_closes[index - 1], "raw_close_after": raw_closes[index]}
        base = {"instrument_id": instrument_id, "ex_date": step["ex_date"], "yahoo_symbol": ticker,
                "observed_step": step["step"], "observed_dispersion": step["dispersion"]}

        if event_date is not None:
            matched_events.add(event_date)
            # Inside a hole in the raw bars the step can only be seen at the first bar after it,
            # while the event names the day itself: flattrade has no BSE RELIANCE bars from
            # 2024-10-20 to 2024-11-20, across the 2024-10-28 bonus.
            if days[index - 1] < event_date <= days[index]:
                base["ex_date"] = event_date
            ratio = events[event_date]
            expected = 1 / ratio
            shares_after, shares_before = exact_ratio(ratio)
            residual = step["step"] / expected
            status = "confirmed" if residual <= 1 + EVENT_TOLERANCE else "provisional"
            rows.append(dict(base, kind="split", shares_after=shares_after, shares_before=shares_before,
                             price_factor=shares_before / shares_after, volume_factor=shares_after / shares_before,
                             status=status, source="yahoo_split_event", yahoo_event_date=event_date,
                             yahoo_event_ratio=ratio, evidence=dict(evidence, residual=round(residual, 6))))
            if residual < 1 - EVENT_TOLERANCE:
                rows.append(dict(base, kind="unclassified", shares_after=None, shares_before=None,
                                 price_factor=residual, volume_factor=1, status="provisional",
                                 source="yahoo_close_ratio", yahoo_event_date=None, yahoo_event_ratio=None,
                                 evidence=dict(evidence, note="remainder after the listed split")))
            continue

        # A step Yahoo lists no event for is a demerger, a rights issue, or an inconsistency in one
        # of the two price series, and prices alone do not say which: UPL's rights issue of
        # 2024-11-26 and RELIANCE's Jio Financial demerger of 2023-07-20 look the same here. So it
        # waits for a person, who confirms a demerger by setting its kind and status.
        raw_move = raw_closes[index] / raw_closes[index - 1]
        demerger_like = (DEMERGER_RANGE[0] <= step["step"] <= DEMERGER_RANGE[1]
                         and abs(raw_move - step["step"]) <= DEMERGER_GAP_TOLERANCE)
        rows.append(dict(base, kind="unclassified", shares_after=None, shares_before=None,
                         price_factor=step["step"], volume_factor=1, status="provisional",
                         source="yahoo_close_ratio", yahoo_event_date=None, yahoo_event_ratio=None,
                         evidence=dict(evidence, raw_move=round(raw_move, 6), raw_price_fell_alike=demerger_like)))

    for event_date, ratio in events.items():
        if event_date in matched_events:
            continue
        shares_after, shares_before = exact_ratio(ratio)
        inside = days[PERSISTENCE] <= event_date <= days[-PERSISTENCE]
        rows.append({"instrument_id": instrument_id, "ex_date": event_date, "kind": "split",
                     "shares_after": shares_after, "shares_before": shares_before,
                     "price_factor": shares_before / shares_after, "volume_factor": shares_after / shares_before,
                     "status": "rejected" if inside else "provisional", "source": "yahoo_split_event",
                     "yahoo_symbol": ticker, "yahoo_event_date": event_date, "yahoo_event_ratio": ratio,
                     "observed_step": None, "observed_dispersion": None,
                     "evidence": {"note": "listed by Yahoo, but Yahoo's Close and the stored prices move "
                                          "together across it: either there was no action, or the stored "
                                          "prices already include it" if inside
                                  else "listed by Yahoo before the stored history begins; not verifiable"
                                  if event_date < days[PERSISTENCE]
                                  else "listed by Yahoo too recently to check against the stored prices"}})
    return rows

raw_gaps ¶

raw_gaps(instrument_id, exchange, days, raw_closes, source_ids, derived)

Overnight raw moves too large to be trading, with no factor near them.

Parameters:

Name	Type	Description	Default
`instrument_id`	`str`	The instrument.	required
`exchange`	`str`	"nse" or "bse".	required
`days`	`list`	All raw dates, oldest first.	required
`raw_closes`	`list[float]`	Raw closes.	required
`source_ids`	`list[int]`	The source of each bar.	required
`derived`	`list[dict]`	The rows already derived, which cover gaps near them.	required

Returns:

Type	Description
`list[dict]`	Provisional raw_gap rows.

Example usage

From example_2_steps_yahoo_does_not_explain.py

return self.builder.raw_gaps('INDIAGLYCO-NSE', 'nse', days, raw_closes, source_ids, [])

Source code in stock_brokers/instruments/historical/utilities/unified/factors.py

def raw_gaps(self, instrument_id, exchange, days, raw_closes, source_ids, derived):
    """
    Overnight raw moves too large to be trading, with no factor near them.

    Args:
        instrument_id (str): The instrument.
        exchange (str): "nse" or "bse".
        days (list): All raw dates, oldest first.
        raw_closes (list[float]): Raw closes.
        source_ids (list[int]): The source of each bar.
        derived (list[dict]): The rows already derived, which cover gaps near them.

    Returns:
        list[dict]: Provisional raw_gap rows.
    """
    explained = [row["ex_date"] for row in derived if row["status"] != "rejected"]
    rows = []
    for index in range(1, len(days)):
        move = raw_closes[index] / raw_closes[index - 1]
        if 1 - RAW_GAP <= move <= 1 / (1 - RAW_GAP):
            continue
        window = days[max(0, index - MATCH_WINDOW):min(len(days), index + MATCH_WINDOW + 1)]
        if any(window[0] <= day <= window[-1] for day in explained):
            continue
        snapped = Fraction(move).limit_denominator(10)
        evidence = {
            "raw_close_before": raw_closes[index - 1], "raw_close_after": raw_closes[index],
            "move": round(move, 6), "nearest_fraction": f"{snapped.numerator}/{snapped.denominator}",
            "token_changed": source_ids[index] != source_ids[index - 1],
            "dhan_factor": self.dhan_factor(instrument_id, exchange, days[index - 1], raw_closes[index - 1]),
        }
        rows.append({"instrument_id": instrument_id, "ex_date": days[index], "kind": "unclassified",
                     "shares_after": None, "shares_before": None, "price_factor": move, "volume_factor": 1,
                     "status": "provisional", "source": "raw_gap", "yahoo_symbol": None,
                     "yahoo_event_date": None, "yahoo_event_ratio": None, "observed_step": move,
                     "observed_dispersion": None, "evidence": evidence})
    return rows

dhan_factor ¶

dhan_factor(instrument_id, exchange, day, raw_close)

The factor dhan's adjusted history implies for a day: its close over the raw close.

Parameters:

Name	Type	Description	Default
`instrument_id`	`str`	The instrument.	required
`exchange`	`str`	"nse" or "bse".	required
`day`	`date`	The last day before the move.	required
`raw_close`	`float`	The raw close that day.	required

Returns:

Type	Description
`float \| None`	The ratio, or None when dhan has no bar for the day.

Example usage

From example_2_steps_yahoo_does_not_explain.py

dhan = self.builder.dhan_factor('INDIAGLYCO-NSE', 'nse', datetime.date(2026, 9, 1), 485.0)

Source code in stock_brokers/instruments/historical/utilities/unified/factors.py

def dhan_factor(self, instrument_id, exchange, day, raw_close):
    """
    The factor dhan's adjusted history implies for a day: its close over the raw close.

    Args:
        instrument_id (str): The instrument.
        exchange (str): "nse" or "bse".
        day (datetime.date): The last day before the move.
        raw_close (float): The raw close that day.

    Returns:
        float | None: The ratio, or None when dhan has no bar for the day.
    """
    segment = "NSE_EQ" if exchange == "nse" else "BSE_EQ"
    with self.connection.cursor() as cursor:
        cursor.execute(f"""
            select p.close from {tables.BROKER_MAPPINGS} b
            join dhan.price_history p
              on p.instrument_token like b.broker_token || '|' || %s || '|%%'
             and p."interval" = 'day'
             and p."time" = (%s::timestamp at time zone 'Asia/Kolkata')
            where b.instrument_id = %s and b.broker = 'dhan'
            order by b.mapping_date desc limit 1
        """, (segment, day, instrument_id))
        row = cursor.fetchone()
    if row is None or row[0] is None:
        return None
    return round(float(row[0]) / raw_close, 6)

store ¶

store(instrument_id, rows)

Upsert derived rows, leaving manual and rejected rows as they are.

Parameters:

Name	Type	Description	Default
`instrument_id`	`str`	The instrument.	required
`rows`	`list[dict]`	The derived rows.	required

Returns:

Type	Description
`None`	This function returns nothing.

Example usage

From example_2_steps_yahoo_does_not_explain.py

self.builder.store('DEMERGE-NSE', derived)

Source code in stock_brokers/instruments/historical/utilities/unified/factors.py

def store(self, instrument_id, rows):
    """
    Upsert derived rows, leaving manual and rejected rows as they are.

    Args:
        instrument_id (str): The instrument.
        rows (list[dict]): The derived rows.

    Returns:
        None: This function returns nothing.
    """
    with self.connection.cursor() as cursor:
        # A date a person has already decided is left to that decision.
        cursor.execute(f"""
            select ex_date from {tables.ADJUSTMENT_FACTORS} where instrument_id = %s and source = 'manual'
        """, (instrument_id,))
        decided = {row[0] for row in cursor.fetchall()}
        rows = [row for row in rows if row["ex_date"] not in decided]
        cursor.execute(f"""
            delete from {tables.ADJUSTMENT_FACTORS}
            where instrument_id = %s and source <> 'manual' and status <> 'rejected'
              and not (ex_date::text || '|' || kind = any(%s))
        """, (instrument_id, [f"{row['ex_date'].isoformat()}|{row['kind']}" for row in rows]))
        self.counts["stale_deleted"] = self.counts.get("stale_deleted", 0) + cursor.rowcount
        for row in rows:
            cursor.execute(f"""
                insert into {tables.ADJUSTMENT_FACTORS}
                    (instrument_id, ex_date, kind, shares_after, shares_before, price_factor, volume_factor,
                     status, source, yahoo_symbol, yahoo_event_date, yahoo_event_ratio, observed_step,
                     observed_dispersion, evidence)
                values (%(instrument_id)s, %(ex_date)s, %(kind)s, %(shares_after)s, %(shares_before)s,
                        %(price_factor)s, %(volume_factor)s, %(status)s, %(source)s, %(yahoo_symbol)s,
                        %(yahoo_event_date)s, %(yahoo_event_ratio)s, %(observed_step)s,
                        %(observed_dispersion)s, %(evidence)s)
                on conflict (instrument_id, ex_date, kind) do update set
                    shares_after = excluded.shares_after, shares_before = excluded.shares_before,
                    price_factor = excluded.price_factor, volume_factor = excluded.volume_factor,
                    status = excluded.status, source = excluded.source, yahoo_symbol = excluded.yahoo_symbol,
                    yahoo_event_date = excluded.yahoo_event_date, yahoo_event_ratio = excluded.yahoo_event_ratio,
                    observed_step = excluded.observed_step, observed_dispersion = excluded.observed_dispersion,
                    evidence = excluded.evidence, updated_at = now()
                where {tables.ADJUSTMENT_FACTORS}.source <> 'manual'
                  and {tables.ADJUSTMENT_FACTORS}.status <> 'rejected'
                  and ({tables.ADJUSTMENT_FACTORS}.price_factor, {tables.ADJUSTMENT_FACTORS}.status,
                       {tables.ADJUSTMENT_FACTORS}.evidence)
                      is distinct from (excluded.price_factor, excluded.status, excluded.evidence)
            """, dict(row, price_factor=round(row["price_factor"], 10), volume_factor=round(row["volume_factor"], 10),
                      observed_step=None if row["observed_step"] is None else round(row["observed_step"], 10),
                      observed_dispersion=None if row["observed_dispersion"] is None
                      else round(row["observed_dispersion"], 10),
                      evidence=json.dumps(row["evidence"], default=str)))
            self.count(f"{row['kind']}_{row['status']}")
    self.connection.commit()

record_fetch ¶

record_fetch(instrument_id, ticker, kind, status, error, history_start, history_end, last_event)

Record where the Yahoo fetch got to for an instrument.

Parameters:

Name	Type	Description	Default
`instrument_id`	`str`	The instrument.	required
`ticker`	`str \| None`	The ticker used.	required
`kind`	`str \| None`	"ticker" or "scrip_code".	required
`status`	`str`	"ok", "empty", "mismatch" or "error".	required
`error`	`str \| None`	What went wrong.	required
`history_start`	`date \| None`	Yahoo's first date.	required
`history_end`	`date \| None`	Yahoo's last date.	required
`last_event`	`date \| None`	Yahoo's latest listed event.	required

Returns:

Type	Description
`None`	This function returns nothing.

Example usage

From example_2_steps_yahoo_does_not_explain.py

self.builder.record_fetch('INDIAGLYCO-NSE', None, None, 'error', 'Yahoo answered 404', None, None, None)

Source code in stock_brokers/instruments/historical/utilities/unified/factors.py

def record_fetch(self, instrument_id, ticker, kind, status, error, history_start, history_end, last_event):
    """
    Record where the Yahoo fetch got to for an instrument.

    Args:
        instrument_id (str): The instrument.
        ticker (str | None): The ticker used.
        kind (str | None): "ticker" or "scrip_code".
        status (str): "ok", "empty", "mismatch" or "error".
        error (str | None): What went wrong.
        history_start (datetime.date | None): Yahoo's first date.
        history_end (datetime.date | None): Yahoo's last date.
        last_event (datetime.date | None): Yahoo's latest listed event.

    Returns:
        None: This function returns nothing.
    """
    with self.connection.cursor() as cursor:
        cursor.execute(f"""
            insert into {tables.YAHOO_FETCH_STATE}
                (instrument_id, yahoo_symbol, symbol_kind, last_fetched_at, last_status, last_error,
                 history_start, history_end, last_event_date)
            values (%s, %s, %s, now(), %s, %s, %s, %s, %s)
            on conflict (instrument_id) do update set
                yahoo_symbol = coalesce(excluded.yahoo_symbol, {tables.YAHOO_FETCH_STATE}.yahoo_symbol),
                symbol_kind = coalesce(excluded.symbol_kind, {tables.YAHOO_FETCH_STATE}.symbol_kind),
                last_fetched_at = excluded.last_fetched_at, last_status = excluded.last_status,
                last_error = excluded.last_error,
                history_start = coalesce(excluded.history_start, {tables.YAHOO_FETCH_STATE}.history_start),
                history_end = coalesce(excluded.history_end, {tables.YAHOO_FETCH_STATE}.history_end),
                last_event_date = coalesce(excluded.last_event_date, {tables.YAHOO_FETCH_STATE}.last_event_date)
        """, (instrument_id, ticker, kind, status, error, history_start, history_end, last_event))
    self.connection.commit()

tolerance ¶

tolerance(raw_close, level)

How far q may wander from its level without being a move, for a given price.

Parameters:

Name	Type	Description	Default
`raw_close`	`float`	The raw close that day.	required
`level`	`float`	The current level of q.	required

Returns:

Type	Description
`float`	A relative tolerance.

Source code in stock_brokers/instruments/historical/utilities/unified/factors.py

def tolerance(raw_close, level):
    """
    How far q may wander from its level without being a move, for a given price.

    Args:
        raw_close (float): The raw close that day.
        level (float): The current level of q.

    Returns:
        float: A relative tolerance.
    """
    adjusted = max(raw_close * level, 0.01)
    return 0.003 + 0.01 / adjusted

find_steps ¶

find_steps(days, ratios, raw_closes)

The persistent level changes in q, newest first.

Parameters:

Name	Type	Description	Default
`days`	`list[date]`	Overlap dates, oldest first.	required
`ratios`	`list[float]`	q on each date.	required
`raw_closes`	`list[float]`	The raw close on each date.	required

Returns:

Type	Description
`list[dict]`	Each with "index" (of the first day at the newer level), "ex_date", "step" (older level over newer level) and "dispersion".

Source code in stock_brokers/instruments/historical/utilities/unified/factors.py

def find_steps(days, ratios, raw_closes):
    """
    The persistent level changes in q, newest first.

    Args:
        days (list[datetime.date]): Overlap dates, oldest first.
        ratios (list[float]): q on each date.
        raw_closes (list[float]): The raw close on each date.

    Returns:
        list[dict]: Each with "index" (of the first day at the newer level), "ex_date", "step" (older
            level over newer level) and "dispersion".
    """
    steps = []
    level = ratios[-1]
    boundary = len(days)
    position = len(days) - 1
    while position >= PERSISTENCE:
        if abs(ratios[position] / level - 1) <= tolerance(raw_closes[position], level):
            position -= 1
            continue
        window = ratios[position - PERSISTENCE + 1:position + 1]
        candidate = median(window)
        persistent = all(abs(value / level - 1) > tolerance(raw_closes[position], level) for value in window)
        consistent = all(abs(value / candidate - 1) <= tolerance(raw_closes[position], candidate) for value in window)
        if not (persistent and consistent):
            position -= 1
            continue
        newer = ratios[position + 1:min(position + 1 + SIDE_WINDOW, boundary)]
        older_start = position
        while older_start > 0 and position - older_start < SIDE_WINDOW and \
                abs(ratios[older_start - 1] / candidate - 1) <= tolerance(raw_closes[older_start - 1], candidate):
            older_start -= 1
        older = ratios[older_start:position + 1]
        newer_level = median(newer) if newer else level
        older_level = median(older)
        dispersion = max(abs(value / older_level - 1) for value in older)
        steps.append({
            "index": position + 1,
            "ex_date": days[position + 1],
            "step": older_level / newer_level,
            "dispersion": dispersion,
        })
        boundary = position + 1
        level = older_level
        position -= PERSISTENCE
    return steps

exact_ratio ¶

exact_ratio(value)

A split ratio as shares after and before.

Parameters:

Name	Type	Description	Default
`value`	`float`	Yahoo's ratio, for example 2.0 or 0.3.	required

Returns:

Type	Description
`tuple[int, int]`	(shares_after, shares_before), for example (2, 1) or (3, 10).

Source code in stock_brokers/instruments/historical/utilities/unified/factors.py

def exact_ratio(value):
    """
    A split ratio as shares after and before.

    Args:
        value (float): Yahoo's ratio, for example 2.0 or 0.3.

    Returns:
        tuple[int, int]: (shares_after, shares_before), for example (2, 1) or (3, 10).
    """
    # A consolidation's ratio is below one and can be small enough, 0.001, to round to 0/1 if
    # approximated directly, so it is approximated through its inverse.
    if value >= 1:
        fraction = Fraction(value).limit_denominator(100)
        return fraction.numerator, fraction.denominator
    fraction = Fraction(1 / value).limit_denominator(100)
    return fraction.denominator, fraction.numerator