Managing private files with Amazon S3

August 10, 2020

There is quite a bit more to Amazon S3 than storing public files. In this article, we look into how we can manage private files. To do so, we learn how to set up a proper private Amazon S3 bucket and how to upload and access files. We use streams and generate presigned URLs with an expiration time.

You can find the code from this series in this repository.

Setting up Amazon S3#

The first thing to do is to create a new bucket.

This time, we intend to restrict access to the files we upload. Every time we want our users to be able to access a file, they will need to do it through our API.

The IAM user that we’ve previously created has access to all our buckets. Therefore, all we need to do to start using it is to add the name of the bucket to our environment variables.

.env#

# ...
AWS_PRIVATE_BUCKET_NAME=nestjs-series-private-bucket

/src/app.module.ts#

An error has occurred. Please try again later.

Managing files through the API#

Once we have the above set up, we can start uploading files to our private bucket. When doing so, we want to save them in a similar way wheKolejny rok n dealing with public files. This time we won’t save the URL of the file, though.

Let’s allow our users to manage some files. To do that, let’s create the entity of a private file. It needs to contain the id of the user.

/src/privateFiles/privateFile.entity.ts#

import { Column, Entity, ManyToOne, PrimaryGeneratedColumn } from "typeorm"
import User from "../users/user.entity"
@Entity()
class PrivateFile {
  @PrimaryGeneratedColumn()
  public id: number
  @Column()
  public key: string
  @ManyToOne(() => User, (owner: User) => owner.files)
  public owner: User
}
export default PrivateFile

Now we need to add information about the other side of the relationship.

/src/users/user.entity.ts#

import { Entity, OneToMany } from "typeorm"
import PrivateFile from "../privateFIles/privateFile.entity"
@Entity()
class User {
  // ...
  @OneToMany(() => PrivateFile, (file: PrivateFile) => file.owner)
  public files: PrivateFile[]
}
export default User

We need to save the key so that we can access or delete our private files. Let’s create a separate service to manage them.

/src/files/privateFiles.service.ts#

import { Injectable } from "@nestjs/common"
import { InjectRepository } from "@nestjs/typeorm"
import { Repository } from "typeorm"
import { S3 } from "aws-sdk"
import { ConfigService } from "@nestjs/config"
import { v4 as uuid } from "uuid"
import PrivateFile from "./privateFile.entity"
@Injectable()
export class PrivateFilesService {
  constructor(
    @InjectRepository(PrivateFile)
    private privateFilesRepository: Repository<PrivateFile>,
    private readonly configService: ConfigService,
  ) {}
  async uploadPrivateFile(dataBuffer: Buffer, ownerId: number, filename: string) {
    const s3 = new S3()
    const uploadResult = await s3
      .upload({
        Bucket: this.configService.get("AWS_PRIVATE_BUCKET_NAME"),
        Body: dataBuffer,
        Key: `${uuid()}-${filename}`,
      })
      .promise()
    const newFile = this.privateFilesRepository.create({
      key: uploadResult.Key,
      owner: {
        id: ownerId,
      },
    })
    await this.privateFilesRepository.save(newFile)
    return newFile
  }
}

/src/files/privateFiles.module.ts#

import { Module } from "@nestjs/common"
import { TypeOrmModule } from "@nestjs/typeorm"
import { PrivateFilesService } from "./privateFiles.service"
import { ConfigModule } from "@nestjs/config"
import PrivateFile from "./privateFile.entity"
@Module({
  imports: [TypeOrmModule.forFeature([PrivateFile]), ConfigModule],
  providers: [PrivateFilesService],
  exports: [PrivateFilesService],
})
export class PrivateFilesModule {}

Once that’s done, we can use all of the above to upload private files for our users.

/src/users/users.service.ts#

import { Injectable } from "@nestjs/common"
import { InjectRepository } from "@nestjs/typeorm"
import { Repository } from "typeorm"
import User from "./user.entity"
import { PrivateFilesService } from "../privateFIles/privateFiles.service"
@Injectable()
export class UsersService {
  constructor(
    @InjectRepository(User)
    private usersRepository: Repository<User>,
    private readonly privateFilesService: PrivateFilesService,
  ) {} // ...
  async addPrivateFile(userId: number, imageBuffer: Buffer, filename: string) {
    return this.privateFilesService.uploadPrivateFile(imageBuffer, userId, filename)
  }
}

/src/users/users.controller.ts#

import { UsersService } from "./users.service"
import { Controller, Post, Req, UploadedFile, UseGuards, UseInterceptors } from "@nestjs/common"
import JwtAuthenticationGuard from "../authentication/jwt-authentication.guard"
import RequestWithUser from "../authentication/requestWithUser.interface"
import { FileInterceptor } from "@nestjs/platform-express"
import { Express } from "express"
@Controller("users")
export class UsersController {
  constructor(private readonly usersService: UsersService) {} // ...
  @Post("files")
  @UseGuards(JwtAuthenticationGuard)
  @UseInterceptors(FileInterceptor("file"))
  async addPrivateFile(@Req() request: RequestWithUser, @UploadedFile() file: Express.Multer.File) {
    return this.usersService.addPrivateFile(request.user.id, file.buffer, file.originalname)
  }
}

After doing all of the above, our users can start uploading private files.

You can also implement deleting the files in a similar way to uploading them. It might be worth checking if the currently logged in user is an owner of the file, though.

Accessing private files#

Since the files we upload above are private, we can’t access them by simply entering a URL. Trying to do so will result in getting an error.

There is more than one way to approach this issue. Let’s start with the most straightforward one.

Fetching the file from Amazon S3 as a stream#

The first solution to the above issue is to send the file through our API. The most fitting way to do that is to pipe a readable stream that we can get from the AWS SDK to our response. Thanks to working directly with streams, we don’t have to download the file into the memory in our server.

The first thing to do is to get a readable stream of data from our Amazon S3 bucket.

/src/privateFiles/privateFiles.service.ts#

import { Injectable } from "@nestjs/common"
import { InjectRepository } from "@nestjs/typeorm"
import { Repository } from "typeorm"
import { S3 } from "aws-sdk"
import { ConfigService } from "@nestjs/config"
import PrivateFile from "./privateFile.entity"
import { NotFoundException } from "@nestjs/common"
@Injectable()
export class PrivateFilesService {
  constructor(
    @InjectRepository(PrivateFile)
    private privateFilesRepository: Repository<PrivateFile>,
    private readonly configService: ConfigService,
  ) {} // ...
  public async getPrivateFile(fileId: number) {
    const s3 = new S3()
    const fileInfo = await this.privateFilesRepository.findOne(
      { id: fileId },
      { relations: ["owner"] },
    )
    if (fileInfo) {
      const stream = await s3
        .getObject({
          Bucket: this.configService.get("AWS_PRIVATE_BUCKET_NAME"),
          Key: fileInfo.key,
        })
        .createReadStream()
      return {
        stream,
        info: fileInfo,
      }
    }
    throw new NotFoundException()
  }
}

Now we need to make sure if the users should be able to download the file.

/src/users/users.service.ts#

import { Injectable, UnauthorizedException } from "@nestjs/common"
import { InjectRepository } from "@nestjs/typeorm"
import { Repository } from "typeorm"
import User from "./user.entity"
import { FilesService } from "../files/files.service"
import { PrivateFilesService } from "../privateFIles/privateFiles.service"
@Injectable()
export class UsersService {
  constructor(
    @InjectRepository(User)
    private usersRepository: Repository<User>,
    private readonly filesService: FilesService,
    private readonly privateFilesService: PrivateFilesService,
  ) {} // ...
  async getPrivateFile(userId: number, fileId: number) {
    const file = await this.privateFilesService.getPrivateFile(fileId)
    if (file.info.owner.id === userId) {
      return file
    }
    throw new UnauthorizedException()
  }
}

The interesting thing happens in the controller. Since we are working with streams directly, we need to access the Response object that NestJS uses under the hood.

/src/users/users.controller.ts#

import { UsersService } from "./users.service"
import { Controller, Get, Param, Req, Res, UseGuards } from "@nestjs/common"
import JwtAuthenticationGuard from "../authentication/jwt-authentication.guard"
import RequestWithUser from "../authentication/requestWithUser.interface"
import { Response } from "express"
import FindOneParams from "../utils/findOneParams"
@Controller("users")
export class UsersController {
  constructor(private readonly usersService: UsersService) {} // ...
  @Get("files/:id")
  @UseGuards(JwtAuthenticationGuard)
  async getPrivateFile(
    @Req() request: RequestWithUser,
    @Param() { id }: FindOneParams,
    @Res() res: Response,
  ) {
    const file = await this.usersService.getPrivateFile(request.user.id, Number(id))
    file.stream.pipe(res)
  }
}

Generating presigned URLs#

Responding with the data of the file is not the only solution to provide our users with their files. Another way is generating presigned URLs that allow access for a specific expiration time.

We can generate URLs for different actions. To create one for getting a resource, we need to use the getObject operation name:

/src/files/privateFiles.service.ts#

import { Injectable } from "@nestjs/common"
import { InjectRepository } from "@nestjs/typeorm"
import { Repository } from "typeorm"
import { S3 } from "aws-sdk"
import { ConfigService } from "@nestjs/config"
import PrivateFile from "./privateFile.entity"
@Injectable()
export class PrivateFilesService {
  constructor(
    @InjectRepository(PrivateFile)
    private privateFilesRepository: Repository<PrivateFile>,
    private readonly configService: ConfigService,
  ) {} // ...
  public async generatePresignedUrl(key: string) {
    const s3 = new S3()
    return s3.getSignedUrlPromise("getObject", {
      Bucket: this.configService.get("AWS_PRIVATE_BUCKET_NAME"),
      Key: key,
    })
  }
}

The default expiration time of a presigned URL is 15 minutes. We could change it by adding an Expires parameter.

Let’s provide the user with an array of all of the uploaded files.

/src/users/users.service.ts#

import { Injectable, NotFoundException } from "@nestjs/common"
import { InjectRepository } from "@nestjs/typeorm"
import { Repository } from "typeorm"
import User from "./user.entity"
import { FilesService } from "../files/files.service"
import { PrivateFilesService } from "../privateFIles/privateFiles.service"
@Injectable()
export class UsersService {
  constructor(
    @InjectRepository(User)
    private usersRepository: Repository<User>,
    private readonly filesService: FilesService,
    private readonly privateFilesService: PrivateFilesService,
  ) {} // ...
  async getAllPrivateFiles(userId: number) {
    const userWithFiles = await this.usersRepository.findOne(
      { id: userId },
      { relations: ["files"] },
    )
    if (userWithFiles) {
      return Promise.all(
        userWithFiles.files.map(async (file) => {
          const url = await this.privateFilesService.generatePresignedUrl(file.key)
          return {
            ...file,
            url,
          }
        }),
      )
    }
    throw new NotFoundException("User with this id does not exist")
  }
}

/src/users/users.controller.ts#

import { UsersService } from "./users.service"
import { Controller, Get, Req, UseGuards } from "@nestjs/common"
import JwtAuthenticationGuard from "../authentication/jwt-authentication.guard"
import RequestWithUser from "../authentication/requestWithUser.interface"
@Controller("users")
export class UsersController {
  constructor(private readonly usersService: UsersService) {} // ...
  @Get("files")
  @UseGuards(JwtAuthenticationGuard)
  async getAllPrivateFiles(@Req() request: RequestWithUser) {
    return this.usersService.getAllPrivateFiles(request.user.id)
  }
}

Now, the user can access all of the files in a very straightforward way.

Summary#

In this article, we’ve broadened our knowledge about Amazon S3. This time, we’ve learned how to manage private files. It included fetching them with the use of streams and generating presigned URLs. All of the above knowledge covers a variety of cases. Therefore, it allows us to manage files with Amazon S3 properly.