How to use object storage for large files

· Category: System Design

Short answer

Object storage manages data as objects in flat namespaces, providing virtually unlimited scale and high durability for unstructured content.

Steps

  1. Upload files as objects with unique keys in a bucket or container.
  2. Attach metadata and tags to objects for organization and access control.
  3. Enable versioning to protect against accidental overwrites and deletions.
  4. Configure lifecycle policies to transition old objects to cheaper tiers or delete them.
  5. Serve content directly via HTTP or through a CDN for global delivery.

Tips

  • Use multipart uploads for large files to improve reliability and resume capability.
  • Generate presigned URLs for temporary, secure access to private objects.
  • Encrypt objects at rest and in transit for security compliance.
  • Monitor access logs and set bucket policies to prevent public exposure.

Common issues

  • Latency from frequent small object operations.
  • Eventual consistency in listing operations causing temporary visibility delays.
  • Unexpected egress charges from high download volumes.
  • Lack of fine-grained update capabilities since objects are immutable.

Example

import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(784, 256),
    nn.ReLU(),
    nn.Dropout(0.2),
    nn.Linear(256, 10)
)
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

This snippet defines a simple neural network with dropout for regularization, a cross-entropy loss, and the Adam optimizer in PyTorch.