![]() So, the problem compounds as you index more columns. And a relational database like PostgreSQL keeps a B-tree (or other data structure) for each table index, in order for values in that index to be found efficiently. Now, if the working set of data and indexes is small, we can keep it in memory.īut if the data is sufficiently large that we can’t fit all (similarly fixed-size) pages of our B-tree in memory, then updating a random part of the tree can involve significant disk I/O as we read pages from disk into memory, modify in memory, and then write back out to disk (when evicted to make room for other B-tree pages). With an index, a query can quickly find a row with a specified ID (e.g., bank account number) without scanning the entire table or “walking” the table in some sorted order. Under most relational databases, a table is stored as a collection of fixed-size pages of data (e.g., 8KB pages in PostgreSQL), on top of which the system builds data structures (such as B-trees) to index the data. This is an old, common problem for relational databases. Eventually, our entire dataset will not fit in memory, which is why we’ll need to write our data and indexes to disk. While memory is faster than disk, it is much more expensive: about 20x costlier than solid-state storage like Flash, 100x more expensive than hard drives. Why databases do not normally scale up well: Swapping in/out of memory is expensiveĪ common problem with scaling database performance on a single machine is the significant cost/performance trade-off between memory and disk. Our motivations are twofold: for anyone facing similar problems, to share what we’ve learned and for those considering using TimescaleDB for time-series data (including the skeptics!), to explain some of our design decisions. How time-series data is unique, how one can leverage those differences to overcome the scaling problem, and some performance results.How LSM trees (typically used in NoSQL databases) do not adequately solve the needs of many time-series applications. ![]() ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |