How To Migrate From Catalog to Catalog V2 and Reference Data

This guide will help you migrate from using catalog ("Catalog V1") to catalog-v2 ("Catalog V2") and reference-data ("Reference Data"). Catalog V1 contains both static metadata (name, category, product, etc.) and coverage for a given data type (e.g. min_time and max_time for trades). Due to growing complexity in data coverage and the resulting performance bottlenecks from surfacing all of this data, this metadata is being separated. Catalog V2 and Reference Data allow for more lightweight and flexible queries as they can be queried across several dimensions (for example, catalog-v2/asset-metrics lets you filter by both asset and metric).

Switching between Catalog to Catalog V2 requires a subtle change in how to think about and find the data you're looking for.

In general:

• Use catalog-v2 to get information on when a data type is present.

• Use reference-data to get information on which entities are available and descriptions of what they are.

• catalog-v2 and reference-data take less time to retrieve data per call and go through pagination (which the Python API Client automatically takes care of).

For more information on the reasoning behind the catalog migration, see Catalog V1 to Catalog V2 Migration

Examples

How do I get the assets/markets/exchanges covered?

V1:

• Use catalog/assets, catalog/markets, or catalog/exchanges respectively

V2:

• Use reference-data/assets, reference-data/markets, or reference-data/exchanges respectively

V2

assets = client.reference_data_assets().to_list()
markets = client.reference_data_markets().to_list()
exchanges = client.reference_data_exchanges().to_list()

V1

assets = client.catalog_assets()
markets = client.catalog_markets()
exchanges = client.catalog_exchanges()

How do I get which assets have been on an exchange?

V1: Use catalog/assets and get the "exchanges" field. Note that due to the size of the response, this might take a while.

V2: Use reference-data/markets?asset=<ASSET> and get the "exchange" field for each observation, loop through each observation. (Note that due to pagination, you will have to loop through the entire response).

V2

list_markets = client.reference_data_markets(asset="btc").to_list()
list_exchanges = set([market['exchange'] for market in list_markets])

V1

list_exchanges = client.catalog_assets(assets="btc")[0]['exchanges']

Note that these responses includes exchanges or markets that are now defunct. To get a more precise real-time coverage, see How do I get which the min and max times an asset has been traded in an exchange?

How do I get which markets are on an exchange?

V1: Use catalog/exchanges and get "markets" field

V2: Use reference-data/markets?exchange=<EXCHANGE> and get the "market" field for each observation, loop through each observation. (Note that due to pagination, you will have to loop through the entire response).

V2

list_markets = client.reference_data_markets(exchange="coinbase").to_list()
list_markets = set([market['market'] for market in list_markets])

V1

list_exchanges = client.catalog_exchanges(exchanges="coinbase")[0]['markets']

How do I get the markets for which an asset is a base/quote?

V1: For getting markets where the asset is only one of a base or quote, use catalog/markets?base=<ASSET> or catalog/markets?quote=<ASSET> respectively. For getting markets where the asset is either a base or quote, use catalog/markets?asset=<ASSET>.

V2: For getting markets where the asset is only one of a base or quote, use reference-data/markets?base=<ASSET> or reference-data/markets?quote=<ASSET> respectively. For getting markets where the asset is either a base or quote, use reference-data/markets?asset=<ASSET>.

V2

list_markets = client.reference_data_markets(base="<ASSET>").to_list()

V1

list_markets = client.catalog_markets(base='btc')

How do I get which metrics are covered for a given asset/pair/exchange/market/exchange-asset/institution?

V1: Usecatalog/*-metrics respectively and pass asset, pair, exchange, or market where applicable. Then, loop through each element to search for the presence of that asset, pair, exchange, or market.

V2: Usecatalog-v2/*-metrics respectively and pass asset, pair, exchange, or market where applicable.

Suppose you wanted to know which asset-metrics are available for btc:

V2

list_asset_metrics = client.catalog_asset_metrics_v2(assets='btc').to_list()[0]['metrics']

V1

list_asset_metrics = client.catalog_asset_metrics(assets='btc').to_list()
list_asset_metrics = [metric for metric in list_asset_metrics if 'btc' in metric['frequencies'][0]['assets']]

How do I get which assets/pairs/exchanges/markets are covered for a given metric?

V1: Use catalog/*-metrics and pass your metric(s). Each element will have a frequencies value with a nested assets value for each frequency. Find the appropriate frequency and grab that list of assets.

V2: Use catalog-v2/*-metrics and pass your metric(s). Loop through element and get the asset value.

Suppose you wanted to know what assets are covered by PriceUSD:

V2

list_asset_metrics = client.catalog_asset_metrics_v2(metrics="PriceUSD").to_dataframe()
list_assets = list(df_asset_metrics.loc[((df_asset_metrics.metric=='PriceUSD') & (df_asset_metrics.frequency=='1d')), 'asset'])

V1

list_asset_metrics = client.catalog_asset_metrics(metrics="PriceUSD").to_list()
list_assets = list_asset_metrics[0]['frequencies'][-1]['assets']

How do I get which raw observation data are covered for a given market?

V1 and V2: Use catalog/market-* and catalog-v2/market-* respectively and pass markets where applicable.

Below is an example of using market-trades:

V2

list_markets = client.catalog_market_trades_v2(markets='<MARKET>').to_list()

V1

list_markets = client.catalog_market_trades(markets='<MARKET>')

How do I get min and max time for a market?

V1 and V2: Use catalog/market-trades?asset=<ASSET>&exchange=<EXCHANGE> and catalog-v2/market-trades?asset=<ASSET>&exchange=<EXCHANGE> respectively and get the min and max times.

V2

trades = next(client.catalog_market_trades_v2(markets='coinbase-btc-usd-spot'))
min_time = trades['min_time']
max_time = trades['max_time']

V1

trades = client.catalog_market_trades(markets='coinbase-btc-usd-spot'))
min_time = trades[0]['min_time']
max_time = trades[0]['max_time']

How do I get which the min and max times an asset has been traded in an exchange?

V1 and V2: Use catalog/market-trades?asset=<ASSET>&exchange=<EXCHANGE> and catalog-v2/market-trades?asset=<ASSET>&exchange=<EXCHANGE> respectively and get the min and max times.

V2

trades = client.catalog_market_trades_v2(exchange='coinbase', asset='btc').to_list()
min_time = min([market['min_time'] for market in trades])
max_time = max([market['max_time'] for market in trades])

V1

trades = client.catalog_market_trades(exchange='coinbase', asset='btc')
min_time = trades[0]['min_time']
max_time = trades[0]['max_time']

How do I get what metrics are available for a given product and frequency?

Suppose we wanted to get metrics for the Network Data product that are available at 1 hour frequency.

V1:

• Use catalog/asset-metrics, filter by frequency = '1h' and 'product' = 'Network Data'

V2:

• query reference-data/asset-metrics, filter for 'product' = 'Network Data'

• query catalog-v2/asset-metrics?metrics=<LIST_OF_NETWORK_DATA_METRICS>, filter for frequency = '1h'

V2

asset_metrics_reference = client.reference_data_asset_metrics().to_dataframe()
list_metrics_nd = list(asset_metrics_reference.loc[asset_metrics_reference['product']=='Network Data', 'metric'])

asset_metrics_catalog = client.catalog_asset_metrics_v2(metrics=list_metrics_nd).to_dataframe()
list_hourly_metrics_nd = list(asset_metrics_catalog.loc[asset_metrics_catalog.frequency=='1h', 'metric'].unique())

V1

df_catalog_asset_metrics = client.catalog_asset_metrics().to_dataframe()
list_hourly_metrics_nd = list(df_catalog_asset_metrics.loc[
    (df_catalog_asset_metrics['frequency']=='1h') & (df_catalog_asset_metrics['product']=='Network Data')
])

A Mapping of Catalog Endpoints to Catalog V2 and Reference Data Endpoints

Existing Catalog Endpoint	Catalog v2 Endpoints	Reference Data Endpoints*	Notes
/catalog/assets	/catalog-v2/asset-metrics	/reference-data/assets, /reference-data/markets
/catalog/metrics	/catalog-v2/asset-metrics, /catalog-v2/exchange-metrics, /catalog-v2/exchange-asset-metrics, /catalog-v2/pair-metrics, /catalog-v2/institution-metrics, /catalog-v2/market-metrics	/reference-data/asset-metrics, /reference-data/exchange-metrics, /reference-data/exchange-asset-metrics, /reference-data/pair-metrics, /reference-data/institution-metrics, /reference-data/market-metrics
/catalog/asset-metrics	/catalog-v2/asset-metrics	/reference-data/asset-metrics
/catalog/exchange-metrics	/catalog-v2/exchange-metrics	/reference-data/exchange-metrics
/catalog/exchange-asset-metrics	/catalog-v2/exchange-asset-metrics	/reference-data/exchange-asset-metrics
/catalog/pair-metrics	/catalog-v2/pair-metrics	/reference-data/pair-metrics
/catalog/institution-metrics	/catalog-v2/institution-metrics	/reference-data/institution-metrics
/catalog/exchanges		/reference-data/exchanges, /reference-data/markets
/catalog/exchange-assets	/catalog-v2/exchange-asset-metrics	/reference-data/exchange-asset-metrics
/catalog/pairs	/catalog-v2/pair-metrics	/reference-data/pair-metrics
/catalog/pair-candles	/catalog-v2/pair-candles
/catalog/institutions	/catalog-v2/institution-metrics	/reference-data/institution-metrics
/catalog/markets	/catalog-v2/market-trades, /catalog-v2/market-orderbooks, /catalog-v2/market-quotes, /catalog-v2/market-funding-rates, /catalog-v2/market-open-interest, /catalog-v2/market-liquidations	/reference-data/markets	The existing catalog endpoint will be restricted to 170,000 markets.
/catalog/market-trades	/catalog-v2/market-trades		The existing catalog endpoint will be restricted to 170,000 markets.
/catalog/market-candles	/catalog-v2/market-candles		The existing catalog endpoint will be restricted to 170,000 markets.
/catalog/market-orderbooks	/catalog-v2/market-orderbooks		The existing catalog endpoint will be restricted to 170,000 markets.
/catalog/market-quotes	/catalog-v2/market-quotes		The existing catalog endpoint will be restricted to 170,000 markets.
/catalog/market-funding-rates	/catalog-v2/market-funding-rates		The existing catalog endpoint will be restricted to 170,000 markets.
/catalog/market-contract-prices	/catalog-v2/market-contract-prices		The existing catalog endpoint will be restricted to 170,000 markets.
/catalog/market-implied-volatility	/catalog-v2/market-implied-volatility		The existing catalog endpoint will be restricted to 170,000 markets.
/catalog/market-greeks	/catalog-v2/market-greeks		The existing catalog endpoint will be restricted to 170,000 markets.
/catalog/market-openinterest	/catalog-v2/market-openinterest		The existing catalog endpoint will be restricted to 170,000 markets.
/catalog/market-liquidations	/catalog-v2/market-liquidations		The existing catalog endpoint will be restricted to 170,000 markets.
/catalog/market-metrics	/catalog-v2/market-metrics	/reference-data/market-metrics	The existing catalog endpoint will be restricted to 170,000 markets.
/catalog/indexes	/catalog-v2/index-levels, /catalog-v2/index-constituents	/reference-data/indexes
/catalog/index-candles	/catalog-v2/index-candles
/catalog/asset-alerts		/reference-data/asset-alerts
/catalog/asset-chains	/catalog-v2/asset-chains
/catalog/mempool-feerates	/catalog-v2/mempool-feerates
/catalog/minig-pool-tips-summary	/catalog-v2/mining-pool-tips-summary
/catalog/transaction-tracker	/catalog-v2/transaction-tracker

*Note that reference-data endpoints will show all data covered for a given entity, not just those visible in catalog-v2. This contrasts that of the behavior of catalog-v2 showing just the data available with your API key and catalog-v2-all showing all data.