47 - Cache: Concurrent-Safe Memoization

Previous · Next

A Cache<Key, Value, Error> is a keyed memoizer with a bounded capacity, a time-to-live, and, critically, concurrent deduplication. If ten fibers all miss on the same key, the lookup runs once and ten fibers receive the same result.

What you’ll work through

index.test.ts contains four beats: constructing a Cache with capacity and TTL via Cache.make, invalidating a key, reading hit/miss stats via cache.cacheStats, and observing concurrent deduplication — N simultaneous misses on the same key invoke the lookup once.

By the end you should reach for Cache whenever a lookup is expensive and the same key fires concurrently — DB lookups, remote fetches, computed properties.

Diagnose shape before choosing an optimization

Measure first, then match the primitive to the work you actually have:

• Cache repeated results across time. It also provides single-flight behavior when concurrent callers request the same missing key.
• Batch different concurrent keys with Request and RequestResolver.makeBatched when a downstream system supports bulk reads.
• Pool expensive reusable resources such as database connections (step 38).
• Semaphore bounds concurrency when the downstream system has limited capacity (step 38).

These solve different shapes. A cache does not make ten distinct keys into one database query, and batching does not retain results for later callers. Instrument hit rate, batch size, queue time, and downstream latency so the optimization can be evaluated and removed if it does not pay for its complexity.

Concept

1
Cache.make({ capacity, timeToLive, lookup })
2

3
  cache.get(key)
4
    ├── HIT  ──▶ return cached value
5
    └── MISS ──▶ run lookup(key)
6
                 (concurrent misses on the same key share the lookup)
7

8
Eviction:
9
  - capacity reached    ──▶ least-recently-used entry removed
10
  - timeToLive expired  ──▶ entry recomputed on next get

A plain Map<Key, Promise<Value>> does not deduplicate under contention, the check, the fetch, and the write are three separate steps. Cache collapses them.

1
import { Cache, Duration, Effect } from "effect"
2

3
//      ┌─── Effect<Cache<string, number, never>>
4
//      ▼
5
const program = Effect.gen(function* () {
6
  return yield* Cache.make({
7
    capacity: 1000,
8
    timeToLive: Duration.minutes(5),
9
    lookup: (key: string) => Effect.succeed(key.length),
10
  })
11
})

1
//      ┌─── Effect<number, never>
2
//      ▼
3
const length = cache.get("hello")

1
yield* cache.invalidate("config")
2
yield* cache.refresh("config")

1
//      ┌─── Effect<{ readonly hits: number; readonly misses: number; readonly size: number }>
2
//      ▼
3
const stats = cache.cacheStats

1
//      ┌─── Effect<Effect<{ env: string }>>
2
//      ▼
3
const lazyConfig = Effect.cached(Effect.sync(() => ({ env: "production" })))
4

5
const program = Effect.gen(function* () {
6
  const getConfig = yield* lazyConfig
7
  const a = yield* getConfig // computes
8
  const b = yield* getConfig // cached
9
})

1
//      ┌─── Effect<number>     (number of times the lookup ran)
2
//      ▼
3
const program = Effect.gen(function* () {
4
  let calls = 0
5
  const cache = yield* Cache.make({
6
    capacity: 100,
7
    timeToLive: Duration.minutes(5),
8
    lookup: (key: string) =>
9
      Effect.gen(function* () {
10
        calls++
11
        yield* Effect.sleep("5 millis")
12
        return key.length
13
      }),
14
  })
15

16
  yield* Effect.all(
17
    Array.from({ length: 100 }, () => cache.get("hello")),
18
    { concurrency: "unbounded" },
19
  )
20

21
  return calls
22
})
23
// 1, one lookup served 100 fibers

1
import { Cache, Effect, Duration } from "effect"
2

3
const program = Effect.gen(function* () {
4
  let calls = 0
5
  const cache = yield* Cache.make({
6
    capacity: 100,
7
    timeToLive: Duration.seconds(5),
8
    lookup: (key: string) =>
9
      Effect.sync(() => { calls++; return `value-${key}` }),
10
  })
11

12
  yield* cache.get("k")            // calls === 1
13
  yield* cache.get("k")            // calls === 1 (hit)
14
  // ...6 seconds later...
15
  yield* cache.get("k")            // calls === 2 (TTL expired)
16
})

Common Pitfalls

1, Building the Cache inside a function body. Every call constructs a fresh Cache, so nothing is cached. Build it once during service initialization and share it via the closure or a Context Tag.

2, Using Effect.cached for keyed memoization. Effect.cached memoizes a single Effect with no parameters. For keyed memoization, use Cache.

3, Forgetting that lookup failures are cached. If the lookup fails, the failure is stored until the entry’s TTL expires. Use cache.refresh (which always re-runs the lookup regardless of prior failures) when you want a retry path.

4, Sharing capacity across unrelated keys. A single overflowing key family can evict unrelated entries. Use separate caches for unrelated workloads.

See Also

• Previous, 46-scope: Cache uses Scope for its internal cleanup.
• Next, 42-queue: distribute work; the dual of deduplicating it.
• Related, 40-synchronized-ref: the lower-level primitive Cache is built on top of.
• Related, 32-retry-and-timeout: pairing cache refresh with retry policies.

Run the tests: pnpm test 47-cache

The full test suite

The index.test.ts for this lesson, verbatim from steps/47-cache/.

1
import { describe, expect, it } from "vite-plus/test"
2
import { Cache, Effect, Ref } from "effect"
3

4
describe("Cache", () => {
5
  it("caches by key", async () => {
6
    const program = Effect.gen(function*() {
7
      const calls = yield* Ref.make(0)
8
      const cache = yield* Cache.make<string, number>({
9
        capacity: 100,
10
        lookup: (key) => Ref.update(calls, (n) => n + 1).pipe(Effect.as(key.length)),
11
      })
12
      const first = yield* Cache.get(cache, "effect")
13
      const second = yield* Cache.get(cache, "effect")
14
      return { first, second, calls: yield* Ref.get(calls) }
15
    })
16
    expect(await Effect.runPromise(program)).toEqual({ first: 6, second: 6, calls: 1 })
17
  })
18

19
  it("coalesces concurrent misses into single-flight work", async () => {
20
    const program = Effect.gen(function*() {
21
      const calls = yield* Ref.make(0)
22
      const cache = yield* Cache.make<string, string>({
23
        capacity: 10,
24
        lookup: (key) => Ref.update(calls, (n) => n + 1).pipe(Effect.andThen(Effect.succeed(key))),
25
      })
26
      yield* Effect.all([Cache.get(cache, "same"), Cache.get(cache, "same")], { concurrency: 2 })
27
      return yield* Ref.get(calls)
28
    })
29
    expect(await Effect.runPromise(program)).toBe(1)
30
  })
31
})