- Book: Hexagonal Architecture in Go
- Also by me: Pocket Guides for Developers — the 8-book series
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
Your storage port looks clean: Save(ctx, key string, data []byte) error. The domain never touches the filesystem, the S3 adapter sits behind the interface, and the tests use a map. Then someone uploads a 2 GB video. To call Save, the HTTP handler has to produce a []byte, so it runs io.ReadAll(r.Body).
Read what io.ReadAll does: it builds "slices of exponentially growing size, then copy into a perfectly-sized slice at the end." For a moment, the chunks and the final slice are both alive. That 2 GB upload costs you roughly twice its size in heap before your domain code runs a single line. Under a memory limit of a few gigabytes, two uploads at the same time can be enough to get the container OOM-killed.
Blame the signature. Change it and the memory problem goes away for every adapter at once.
The port: io.Reader in, io.ReadCloser out
The fix is to let bytes flow through the port instead of sitting in it. Put takes an io.Reader and reads it to EOF. Get hands back an io.ReadCloser the caller streams and closes.
package storage import ( "context" "errors" "io" ) var ( ErrNotFound = errors.New("storage: not found") ErrInvalidKey = errors.New("storage: invalid key") ) // Store keeps blobs by key. Put reads r to EOF. // A Put that returns an error must leave no object. type Store interface { Put(ctx context.Context, key string, r io.Reader) (int64, error) Get(ctx context.Context, key string) (io.ReadCloser, error) Delete(ctx context.Context, key string) error } The doc comment on Store is part of the port. "A failed Put leaves no object" is a promise every adapter has to keep, and later a test will hold each one to it.
The handler gets shorter too. http.MaxBytesReader caps the body and still returns a reader, so the cap costs no memory:
func (h Handler) upload(w http.ResponseWriter, r *http.Request) { body := http.MaxBytesReader(w, r.Body, 5<<30) key := "uploads/" + r.PathValue("id") _, err := h.store.Put(r.Context(), key, body) if err != nil { http.Error(w, "upload failed", http.StatusBadRequest) return } w.WriteHeader(http.StatusCreated) } In a real service the handler calls a use case and the use case calls Store. The signature stays the same all the way down: the use case passes the reader through and never buffers it.
The snippets below share one package so they stay short. In a real layout the port sits in storage, each adapter gets its own package (storage/disk, storage/s3store, storage/memory), and the contract test lives in storage/storagetest as an exported RunContract. Only the S3 adapter's package imports the AWS SDK, so the domain never does.
A []byte port fills the bucket first. An io.Reader port is a pipe.
Disk adapter: write a temp file, then rename
The local adapter has one job beyond copying bytes: a reader must never open a half-written file. os.CreateTemp in the same directory as the target, then os.Rename onto the final name. Same directory means same filesystem, so on Unix the rename swaps the name in one step. The Go docs are blunt about the exception: "on non-Unix platforms Rename is not an atomic operation."
First, turning a key into a path:
type Disk struct{ root string } func NewDisk(root string) *Disk { return &Disk{root: root} } func (d *Disk) path(key string) (string, error) { k := filepath.FromSlash(key) if !filepath.IsLocal(k) || filepath.Clean(k) != k || k == "." { return "", ErrInvalidKey } return filepath.Join(d.root, k), nil } The key check uses filepath.IsLocal (Go 1.20+), which rejects absolute paths and anything that climbs out of the root. The extra Clean and "." checks reject keys like uploads/.. that stay inside the root but resolve to a directory. The docs warn it "is a purely lexical operation" that ignores symlinks, so don't let untrusted users create symlinks under the root.
Then Put, which writes to a temp file and only renames it once everything succeeded:
func (d *Disk) Put(ctx context.Context, key string, r io.Reader) (int64, error) { dst, err := d.path(key) if err != nil { return 0, err } dir := filepath.Dir(dst) if err := os.MkdirAll(dir, 0o755); err != nil { return 0, err } tmp, err := os.CreateTemp(dir, ".put-*") if err != nil { return 0, err } done := false defer func() { if !done { os.Remove(tmp.Name()) } }() n, err := io.Copy(tmp, ctxReader{ctx, r}) if err == nil { err = tmp.Chmod(0o644) } if err == nil { err = tmp.Sync() } if cerr := tmp.Close(); err == nil { err = cerr } if err != nil { return n, err } if err := os.Rename(tmp.Name(), dst); err != nil { return n, err } done = true return n, nil } CreateTemp makes the file "with mode 0o600 (before umask)", per the os docs, so without the Chmod your web server may not be able to read what you stored. Sync flushes the data before the name points at it. Close returns an error of its own, so the code checks it instead of dropping it. The deferred cleanup removes the temp file on every failure path.
If the process dies mid-upload, though, a .put-* file stays behind under a name no Get will ask for, so sweep old ones at startup. And if the rename itself has to survive a power cut, also sync the parent directory.
Get and Delete are short. Return the *os.File only on success, so you never hand back a typed nil inside an interface:
func (d *Disk) Get(ctx context.Context, key string) (io.ReadCloser, error) { p, err := d.path(key) if err != nil { return nil, err } f, err := os.Open(p) if errors.Is(err, fs.ErrNotExist) { return nil, ErrNotFound } if err != nil { return nil, err } return f, nil } func (d *Disk) Delete(ctx context.Context, key string) error { p, err := d.path(key) if err != nil { return err } err = os.Remove(p) if errors.Is(err, fs.ErrNotExist) { return nil } return err } The last piece is the reader wrapper Put copies through:
type ctxReader struct { ctx context.Context r io.Reader } func (c ctxReader) Read(p []byte) (int, error) { if err := c.ctx.Err(); err != nil { return 0, err } return c.r.Read(p) } ctxReader stops a copy when the request is cancelled. It also hides any WriterTo on the source, so the copy ends up on io.Copy's fixed 32 * 1024 byte buffer. That buffer is the only per-upload memory this adapter needs.
S3 adapter: transfermanager, not the old upload manager
If your S3 code still imports feature/s3/manager, it's on a deprecated module. The aws-sdk-go-v2 release of 2026-02-05 reads: "Deprecate feature/s3/manager, superceded by feature/s3/transfermanager." The maintainers' announcement puts the successor's production release on January 30, 2026. New code should use transfermanager:
type S3 struct { client *s3.Client tm *transfermanager.Client bucket string } func NewS3(c *s3.Client, bucket string) *S3 { tm := transfermanager.New(c, func(o *transfermanager.Options) { o.FailTimeout = 30 * time.Second }) return &S3{client: c, tm: tm, bucket: bucket} } Put hands the reader to the uploader, wrapped so it can report how many bytes went through:
func (s *S3) Put(ctx context.Context, key string, r io.Reader) (int64, error) { cr := &countingReader{r: r} _, err := s.tm.UploadObject(ctx, &transfermanager.UploadObjectInput{ Bucket: aws.String(s.bucket), Key: aws.String(key), Body: cr, }) return cr.n, err } type countingReader struct { r io.Reader n int64 } func (c *countingReader) Read(p []byte) (int, error) { n, err := c.r.Read(p) c.n += int64(n) return n, err } What does "streaming" mean for an HTTP body you can't seek? Reading the UploadObject source, it reads up to the multipart threshold into memory first. If the body ends before that, it sends one PutObject. Otherwise it pulls parts from a pool of Concurrency+1 buffers of PartSizeBytes each. The defaults in api_client.go are a 16 MiB threshold, 8 MiB parts, concurrency 5 and 10,000 max parts. So memory per upload is bounded by those knobs, not by the file size.
Two consequences of those numbers. First, a non-seekable stream at 8 MiB parts tops out at 10,000 parts, about 78 GiB, because the uploader can only grow the part size when it knows the length up front. Raise PartSizeBytes if you accept bigger streams. Second, the failure promise holds. A single PutObject is all or nothing: "Amazon S3 never adds partial objects," per the PutObject reference. On the multipart path, a read or part error makes the uploader call AbortMultipartUpload before returning, and an unfinished multipart upload is never a readable object.
By default the uploader runs that abort on the context you passed in, which here is the request context. The server cancels that context when the client's connection closes, so a client that hangs up can take the abort down with it. The FailTimeout option gives the cleanup a fresh context so it can "succeed even when the original context is canceled," which is why NewS3 sets it. A process that crashes mid-upload can't abort anything at all, and S3 keeps billing for the parts until you complete or stop the upload. Add the AbortIncompleteMultipartUpload lifecycle rule that page recommends as the backstop for both cases.
Get and Delete call the plain S3 client:
func (s *S3) Get(ctx context.Context, key string) (io.ReadCloser, error) { out, err := s.client.GetObject(ctx, &s3.GetObjectInput{ Bucket: aws.String(s.bucket), Key: aws.String(key), }) var nsk *types.NoSuchKey if errors.As(err, &nsk) { return nil, ErrNotFound } if err != nil { return nil, err } return out.Body, nil } func (s *S3) Delete(ctx context.Context, key string) error { _, err := s.client.DeleteObject(ctx, &s3.DeleteObjectInput{ Bucket: aws.String(s.bucket), Key: aws.String(key), }) return err } Imports are aws, feature/s3/transfermanager, service/s3 and service/s3/types from github.com/aws/aws-sdk-go-v2, plus time from the standard library. The errors.As pattern for modeled S3 errors is the one in the SDK's error-handling guide.
One trap on Get: per the GetObject reference, a missing key returns 404 only if the caller has s3:ListBucket. Without it you get 403, and your adapter reports a permissions error where you expected ErrNotFound.
Disk, S3 and the in-memory fake all plug into the same Store interface.
The fake, and one contract test all three must pass
Use-case tests need a store that is fast and has no setup. The fake buffers, which is fine for test-sized data:
type Memory struct { mu sync.Mutex objs map[string][]byte } func NewMemory() *Memory { return &Memory{objs: map[string][]byte{}} } func (m *Memory) Put(ctx context.Context, key string, r io.Reader) (int64, error) { b, err := io.ReadAll(ctxReader{ctx, r}) if err != nil { return int64(len(b)), err } m.mu.Lock() defer m.mu.Unlock() m.objs[key] = b return int64(len(b)), nil } Get hands back a new reader over the stored bytes on every call, so two readers never share a position:
func (m *Memory) Get(ctx context.Context, key string) (io.ReadCloser, error) { m.mu.Lock() defer m.mu.Unlock() b, ok := m.objs[key] if !ok { return nil, ErrNotFound } return io.NopCloser(bytes.NewReader(b)), nil } func (m *Memory) Delete(ctx context.Context, key string) error { m.mu.Lock() defer m.mu.Unlock() delete(m.objs, key) return nil } A fake is only useful if it behaves like the real thing. Write the port's promises down once as a test suite, then run it against every adapter. iotest.ErrReader simulates a client that disconnects halfway:
func RunContract(t *testing.T, newStore func(*testing.T) Store) { ctx := context.Background() t.Run("put then get", func(t *testing.T) { s := newStore(t) n, err := s.Put(ctx, "a/b.txt", strings.NewReader("hello")) if err != nil || n != 5 { t.Fatalf("put: n=%d err=%v", n, err) } if got := read(t, s, "a/b.txt"); got != "hello" { t.Fatalf("got %q", got) Source：https://dev.to/gabrielanhaia/a-file-storage-port-in-go-that-streams-ioreader-in-disk-and-s3-adapters-out-a51Related posts
The Annual Renewal Is a Year-Long Conversation, Not a Surprise
My money had no memory, so I built an agent that does
Give your LangChain agent a leash
I put my clients in the same Discord channel as my AI agents (and Switch stopped them from sneaking in free work)
Find which prospects run Shopify, HubSpot or Microsoft 365, from a list of domains
